Django is set to operate in auto-commit mode by default. Unless a transaction is active, each query is immediately committed to the database. Therefore, it means that by default:
• every query is executed to the database immediately
• each SQL query gets wrapped in its transaction
• every transaction is automatically committed or rolled back
Transactions in Django
Database transactions
A transaction is a sequence of one or more SQL operations treated as a unit. All functions should be executed successfully to call the transaction successful. Ideally, database transactions have what is referred to as the ACID properties.
These properties are four in total and are the standard set of properties usually aimed to guarantee that transactions in the database are processed reliably. One of their duties is to ensure database recoverability in case of any failure.
ACID properties
The following are the building blocks of ACID:
Atomic
Atomicity refers to the ability to ensure that either nothing or everything succeeds when dealing with a transaction.
Consistency
Consistency is the property that fosters that the data will be consistent.
Isolated
No transaction will affect any other transaction by ensuring that every transaction is in isolation.
Durable
Durability advocates for data to remain in the system, even if a system crash immediately following the transaction. All of this should happen as long as a transaction is committed.
Overall, atomicity is the defining property of database transactions. Atomicity requires us to create a block of code within which the atomicity on the database is guaranteed.
There are two ways of implementing a Django atomic solution as follows.
From django.db import transaction # as a decorator @transaction.atomic def task(request): pass # using a context manager with transaction.atomic(): name.object.get()
Django automatically uses transactions or savepoints to ensure the integrity of ORM operations that involve numerous queries, such as remove() and update().
Django’s TestCase class implements a wrapup of each test in a transaction for speed reasons.
Linking Transactions to HTTP requests
Wrapping each request in a transaction is a standard approach to handling transactions on the web. Set ATOMIC_REQUESTS to True when implementing database configurations to sustain this behavior.
First, Django initiates a transaction before invoking the view method. Then, Django commits the transaction if the response is produced without errors. Django rolls back the transaction if the view throws an exception.
Savepoints in your view code, generally using the atomic() context manager, can be used to accomplish subtransactions.
However, after the view, either all or no adjustments are implemented.
Warning
While the convenience of using this transaction mechanism is enticing, it is wasteful as traffic grows. There is some overhead in opening a transaction for each view. The impact on performance is determined by your application’s query patterns and how efficiently your database manages to lock.
Streaming responses and per-request transactions
When a view produces a StreamingHttpResponse, reading the response’s contents triggers code execution to generate the content. Such code runs outside of the transaction because the view has already returned.
In general, it’s not a good idea to write to the database while generating a streaming response because there’s no mechanism to manage issues once the response is sent.
This feature uses the atomic() decorator to wrap every view method in practice.
It’s worth noting that the transactions only include the execution of your view. The rendering of template answers, like middleware, happens outside of the transaction.
It is still feasible to block views from running in a transaction when ATOMIC_REQUESTS is enabled.
non_atomic_requests(using=None) #For a particular view, this decorator will nullify the effect of ATOMIC_REQUESTS: from django.db import transaction @transaction.non_atomic_requests def test_view(request): action () @transaction.non_atomic_requests(using='other') def test_other_view(request): do_stuff_on_the_other_database()
Note that this applies to the view on the condition that it is applied.
Explicitly controlling transactions
To manage database transactions, Django provides a single API.
atomic(using=None, savepoint=True, durable=False)
Database transactions are defined by their atomicity. We can use atomic to generate a block of code that guarantees atomicity on the database. If the code block is completed, the resultant changes will be committed to the database. If an exception occurs, the changes are reversed.
Atomic blocks can be nested to create more complex structures. When an inner block completes successfully in this situation, the consequences of the inner block can, however, be reversed if an exception is triggered in the outer block later.
When an atomic block is always the outermost atomic block, any database changes are committed when the block is exited without issues.
Durability is the term for this, and it is obtained by setting durable=True. If an atomic block is nested inside another, a RuntimeError is thrown.
Atomic can also be used as a decorator.
Even if generate_relationships() breaks an integrity constraint and creates a database error, you can still run queries in add_children() because the changes from create_parent() are still there and connected to the same transaction.
When handle_exception() is called, any activities initiated in generate_relationships() will have been safely rolled back, allowing the exception handler to work on the database if necessary.
Avoid catching exceptions inside atomic transactions
Django considers whether an atomic block was departed typically or except when deciding whether to commit or rollback. You can hide an issue from Django if you catch and handle exceptions inside an atomic block. However, it can lead to strange behavior.
DatabaseError and its subclasses, such as IntegrityError, are particularly vulnerable to this. Django will throw a TransactionManagementError if you conduct database queries before the rollback. After such an error, Django will perform a rollback after the atomic block because the transaction is broken. You may see this behavior when an ORM-related signal handler raises an exception.
The correct technique to catch database problems is around an atomic block, as illustrated above. If necessary, don’t hesitate to have an additional atomic block solely for this reason. A further advantage of this pattern is that it specifies which operations to roll back if an exception occurs.
Django’s behavior is undefined and database-dependent if you catch exceptions raised by raw SQL queries. Thus, you can opt to revert the model state when rolling back a transaction manually.
When a transaction is rolled back, the values of a model’s fields are not reverted. Unless you manually restore the old field values, this could result in an inconsistent model state.
For example, if the transaction fails to update the active field to True, this snippet enforces the use of the correct value in obj.active:
from django.db import DatabaseError, transaction user_obj = UserModel(active=False) user_obj.active = True try: with transaction.atomic(): user_obj.save() except DatabaseError: user_obj.active = False if user_obj.active: …
Some APIs are disabled by atomic to ensure atomicity. Attempting to commit, rollback, or change the database connection’s auto-commit state within an atomic block will result in an exception.
Using a parameter to atomic must be the name of a database. Django uses the “default” database if this argument is not specified.
Django’s transaction management code is as follows:
When entering the outermost atomic block, a transaction is started; when entering an inner atomic block, a savepoint is created; when quitting an inner block, the savepoint is released or rolled back; when exiting the outermost block, the transaction is committed or rolled back.
Setting the savepoint option to False disables the establishment of savepoints for inner blocks. If an exception occurs, Django will roll back to the earliest parent block with a savepoint if one exists or the outermost block if none exists.
The outer transaction still guarantees atomicity. Only use this option if the overhead of savepoints is significant. It has the disadvantage of breaking the error mentioned above handling.
When auto-commit is disabled, you can use atomic. Even for the outermost block, it will only consume savepoints.
Why does Django make use of auto-commit?
According to SQL standards, each SQL query initiates a transaction unless one is already in progress. Such transactions must then be committed or rolled back expressly.
For application developers, this isn’t always practical. Most databases provide an auto-commit setting to help with this issue.
Each SQL query is wrapped in its transaction when auto-commit is enabled and no transaction runs. In other words, each such question not only initiates a transaction but also commits or rolls back the transaction based on whether the query succeeds. Autocommit must be disabled initially, according to PEP 249, the Python Database API Specification However, v2.0. Django overrides the default behavior and enables auto-commit.
You can disable transaction management to avoid this, but this is not recommended.
How to turn off Transaction management?
You can completely deactivate Django’s transaction management by setting AUTOCOMMIT to False in a database’s setup. Django will disable auto-commit and do no commits if you do this. Thus, you’ll obtain the underlying database library’s normal behavior.
It necessitates explicitly committing every transaction, including those initiated by Django or third-party libraries. As a result, this is best used when you need to run your transaction-controlling middleware or perform anything unusual.
Taking actions after a commit
Occasionally, you may need to conduct an action relating to the current database transaction, but only if the transaction has been committed successfully. Examples include a Celery task, an email notification, or cache invalidation.
The on_commit() function in Django allows you to register callback routines that are executed after a transaction has been appropriately committed:
on_commit(func, using=None)
Pass any function to on commit() that doesn’t take any arguments:
from django.db import transaction def do_work(): pass # invalidate a cache, fire off a Celery task or send a mail, etc. transaction.on_commit(do_ work)
Alternatively, wrap the function in a lambda as follows:
transaction.on_commit(lambda: specify_celery_task.delay('user_arg'))
The function you supply here will be invoked immediately after a hypothetical database write is successfully committed using on_commit().
The callback will be executed if you use on_commit() for no ongoing transaction.
Your function will be ignored and never called if the hypothetical database write is instead rolled back. The latter usually happens when an unhandled exception is raised in an atomic() block).
Savepoints
It is a transactional point where you can roll back a portion of it rather than the entire transaction. Oracle, MySQL, SQLite, and PostgreSQL databases support savepoints using the InnoDB storage engine. Other backends include savepoint functions, but they’re just that: they’re empty operations that don’t do anything.
If you’re utilizing Django’s default behavior, auto-commit, savepoints aren’t useful. When you use atomic() to begin a transaction, you create a series of database operations that must be committed or rolled back. The entire transaction is rolled back when you issue a rollback. Savepoints allow you to do a fine-grained rollback rather than the entire rollback that a transaction would give transaction.Rollback().
The atomic() decorator generates a savepoint to allow partial commit or rollback when nested. Although it is strongly recommended to utilize atomic() instead of the functions listed below, they are still part of the public API and will not be deprecated.
Each of these functions has a parameter – the name of the database that the behavior applies to. The “default” database is utilized if no using parameter is specified.
Three functions exist in django.db.transaction control savepoints:
savepoint(using=None)
This function creates a new savepoint. It indicates that the transaction is in “good” status. The savepoint ID is returned.
savepoint_commit(sid, using=None)
Savepoint sid is released. The changes made since the savepoint was created were incorporated into the transaction.
savepoint rollback(sid, using=None)
The function above is a function that saves a point in time, i.e., the transaction is rolled back to savepoint sid. These functions are useless if savepoints aren’t supported or the database is in auto-commit mode.
Furthermore, there is a utility function:
clean savepoints(using=None)
The function is responsible for resetting the counter used to generate unique savepoint IDs. The use of savepoints is demonstrated in the following example:
from django.db import transaction # initializing a transaction @transaction.atomic def viewing_func(request): func_a.save() # as at now, the transaction should now have a.save() sid = transaction.savepoint() func_b.save() # the transaction should now have func_a.save() and func_b.save() if want_to_keep_b: transaction.savepoint_commit(sid) # the open transaction has func_a.save() and func_b.save() else: transaction.savepoint_rollback(sid) # the open transaction only has func_a.save()
By doing a partial rollback, savepoints can recover from a database error. Because it is not aware of your handling the situation at a lower level, the entire block is rolled back if you do this inside an atomic() block! To avoid this, use the following routines to manage the rollback behavior.
get_rollback(using=None) set_rollback(rollback, using=None)
When the rollback flag is set to True, a rollback occurs when the innermost atomic block is exited. It could be handy for causing a rollback without causing an exception to be raised.
Before proceeding, make sure you’ve rolled back the transaction to a known-good savepoint within the current atomic block! If you set it to False, you won’t get a rollback. Otherwise, you’re breaking atomicity, which could lead to data corruption.
execution order
On-commit functions are executed in which they were registered for a specific transaction.
Handling exceptions
If one of the on-commit functions in a transaction throws an uncaught exception, none of the transaction’s following registered functions will run. It is the same as running the routines sequentially without using on commit ().
Timing of execution
Because your callbacks are executed after a successful commit, a callback failure will not result in the transaction being rolled back. They are executed if the transaction is successful, but they are not a part of the transaction. If your follow-up action is so essential that its failure would cause the transaction’s failure, you shouldn’t use the on commit() hook.
It should suffice for the expected use cases, including email notifications, Celery jobs, etc. Instead, you might want to use a two-phase commit supported by the psycopg Two-Phase Commit protocol, and the Python DB-API specification’s optional.
Two-Phase Commit Extensions
Callbacks are not executed until the connection’s auto-commit is restored after the commit. That is because otherwise, any queries done in a callback would open an implicit transaction. As a result, this prevents the connection from going back into auto-commit mode.
The function will run instantly, not on commit, when in auto-commit mode and outside an atomic() block.
Only auto-commit mode and the atomic() (or ATOMIC REQUESTS) transaction API support on-commit routines. Calling on commit() will error if you are not within an atomic block, and auto-commit is additionally disabled.
Incorporating into tests
To offer test isolation, Django’s TestCase class wraps each test in a transaction and rolls back that transaction after each test. It means that your on-commit () callbacks will never be called because no transaction is ever actually committed.
Using TestCase, you can get around this limitation.
captureOnCommitCallbacks() creates a list of your on commit() callbacks, allowing you to make tests against them or simulate the transaction completed by calling them.
Another option is to use TransactionTestCase instead of TestCase to get around the limitation. Your transaction is committed, and the callbacks will be executed. TransactionTestCase, on the other hand, flushes the database between tests, which is much slower than TestCase’s isolation.
Why isn’t there a rollback hook?
Because several causes can cause an implicit rollback, a rollback hook is more difficult to create than a commit hook. Your rollback hook will never trigger if your database connection is terminated. That is because the process was not given time to shut down usually. Instead, the process is killed.
However, instead of performing something during the atomic block (transaction) and then reversing it if the transaction fails, use on_commit() to postpone doing it until after the transaction succeeds. It’s a lot easier to undo something you’ve never done before!
Autocommit
Django’s django.db.transaction module has an API for managing the auto-commit state of each database connection.
get_autocommit(using=None)<br>set_autocommit(autocommit, using=None)
These functions require a using parameter, which should be a database name. Django uses the “default” database if it isn’t given.
Autocommit is enabled by default. You must turn it back on if you turn it off.
When you disable auto-commit, your database adapter’s default behavior takes over, and Django cannot help. Even though PEP 249 specifies this behavior, adapter implementations aren’t always consistent. Carefully read the documentation for the adapter you’re using.
Before turning auto-commit back on, make sure no transactions are active by calling a commit() or a rollback().
When an atomic() block is running, Django will refuse to turn off the auto-commit since it would destroy atomicity.
Transactions
Atomic series of the database queries are referred to as a transaction. Even if your software crashes, the database ensures that all or no modifications are applied.
Django lacks an API for initiating a transaction. Disabling auto-commit using set_autocommit() is the standard way to begin a transaction.
You can use commit() to apply the changes you’ve made so far or roll back to undo them even when you are in the midst of the transaction. Django.db.transaction defines these functions.
commit(using=None) srollback(using=None)
These functions require a using parameter, which should be a database name. Django uses the “default” database if it isn’t given.
When an atomic() block is active, Django will refuse to commit or roll back since it would break atomicity.
Conclusion
Django gives us a few options for managing database transactions. The transaction property is primarily utilized in transaction-related queries. Because Django is run in auto-commit mode, each question is directly committed to the database unless a transaction is active.
Atomic series of database queries are referred to as a transaction.
To govern database transactions, Django provides a single API. Database transactions are defined by their atomicity. As a result, we can use atomic to generate a block of code that guarantees atomicity on the database.
Each SQL query is wrapped in its transaction when auto-commit is enabled, and no transaction runs. In other words, each such question not only initiates a transaction but also commits or rolls back the transaction based on whether the query succeeds.
Django ORM automatically uses transactions or savepoints to ensure the integrity of ORM operations that involve several queries, such as delete() and update().
