In this article, I will show you how you can use DynamoDB as a settings store for your Django application. Some people prefer to store settings in the environment variables of the instance itself, but DynamoDB can be a quite good alternative.
When you don't specify a settings-module for your Django project, the settings.py which is located in your project folder will be used. You can override the settings module in two ways:
--settings=
parameter.DJANGO_SETTINGS_MODULE
environment variable.In the past, I used a separate settings module for each environment, which resulted in multiple settings modules in my codebase:
/project/settings.py
(for local development)
/project/settings_test.py
(for test environment)
/project/settings_prod.py
(for production environment)
I think most people who started developing with Django did it this way initially, however, there are a few drawbacks to this technique:
Disclosure of sensitive information: Keeping settings in your codebase directly, means you also have sensitive information like database usernames and passwords in your codebase.
Subtle changes in test vs production: You add a parameter in your test environment settings module, but you forget to add the parameter in your production settings module.
Changing settings requires deployment: This one speaks for itself. Changing settings should not require a new deployment of your application.
While settings on your local development machine can differ from your production environment (after all, during development we experiment with new things), for actual deployment, we want our test environment to match our production environment as close as possible. In order to make this possible, we need the following three conditions:
Let's see how we can establish those three steps, with a little help from AWS :-)
Since we run our application on EC2 instances, we can use tags to identify our environment. For example, for every instance we launch in our test environment, we can add the following tags:
Environment: test-myapp01
Environment-role: test-myapp01-website
The Environment-role
tag is added to quickly identify EC2 instances when your
application consists of multiple components. For example, you might also have a role
called test-myapp01-mailgateway
if your application sends out email and the mail server
is on a different instance.
If you use tools like CloudFormation or TerraForm (and I really recommend you do), you can have those tags added automatically every time you make a change to your infrastructure.
During startup of our application, we can determine our environment by querying the meta-data of the instance we are running on.
Since our environment is now identified, we can easily load our configuration. I choose DynamoDB as the repository for the application settings, since it's highly-available in your AWS region, it's cheap, and you can manage it via the AWS console.
In this setup I only have two settings modules in my codebase:
/project/settings.py
(for local development)
/project/settings_deploy.py
(for test and production environment)
settings_deploy.py
will retrieve the EC2 tags associated with the instance it is running on,
and retrieve the settings from the DynamoDB table.
DISCLAIMER: This is just a proof-of-concept, and not production-quality code.
The name of the table should be related to the environment-role we run in. For example, if our environment-role is test-myapp-website, we need to create a DynamoDB table that is called test-myapp-website-config.
We will use the AWS command-line tools to do this:
aws dynamodb create-table --table-name=test-myapp-website-config \
--attribute-definitions AttributeName=Parameter,AttributeType=S \
--key-schema AttributeName=Parameter,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1
Next, we will fill this table with some default settings. Create the file
test-myapp-website-config.json
with the following content:
{
"test-myapp-website-config": [
{
"PutRequest": {
"Item": {
"Parameter": {"S": "debug"},
"Value": {"BOOL": true}
}
}
},
{
"PutRequest": {
"Item": {
"Parameter": {"S": "db_host"},
"Value": {"S": "my.test.server"}
}
}
},
{
"PutRequest": {
"Item": {
"Parameter": {"S": "db_name"},
"Value": {"S": "my_db"}
}
}
},
{
"PutRequest": {
"Item": {
"Parameter": {"S": "db_user"},
"Value": {"S": "my_username"}
}
}
},
{
"PutRequest": {
"Item": {
"Parameter": {"S": "db_pass"},
"Value": {"S": "my_password"}
}
}
},
{
"PutRequest": {
"Item": {
"Parameter": {"S": "db_port"},
"Value": {"S": "5432"}
}
}
}
]
}
Next step, load this file into your DynamoDB table:
aws dynamodb batch-write-item --request-items file://test-myapp-website-config.json
Make sure you have Boto and Requests installed:
pip install requests
pip install boto
At the top of our settings_deploy.py
file, we can add the following code to
retrieve the value of our Environment-role
tag:
# get environment
r = requests.get('http://169.254.169.254/latest/meta-data/instance-id')
if r.status_code == requests.codes.ok:
instance_id = r.text
conn = ec2.connect_to_region(AWS_REGION_NAME)
reservations = conn.get_all_instances()
for res in reservations:
for inst in res.instances:
if inst.__dict__['id'] == instance_id:
AWS_ENV = inst.__dict__['tags']['Environment-role']
Now we can construct the name of our table, and connect to it:
dynamo_conn = dynamodb.connect_to_region(AWS_REGION_NAME)
config_table = dynamo_conn.get_table('{}-config'.format(AWS_ENV))
After we connected to the table, we can retrieve our settings:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': config_table.get_item(hash_key='db_name')['Value'],
'USER': config_table.get_item(hash_key='db_user')['Value'],
'PASSWORD': config_table.get_item(hash_key='db_pass')['Value'],
'HOST': config_table.get_item(hash_key='db_host')['Value'],
'PORT': config_table.get_item(hash_key='db_port')['Value'],
}
}
Our instances need some additional permissions, to read the EC2 tags, and read the DynamoDB table. Add the following to your instance's IAM role:
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeTags"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"dynamodb:DescribeTable",
"dynamodb:GetItem",
"dynamodb:BatchGetItem"
],
"Resource": [
"<arn of your dynamoDB table>"
]
}
You can now create a similar table for your production environment, and tag your production instances in the same way.
This is just a quick example, and you might want to do some extra work before you start implementing this:
BatchGetItem
to retrieve all settings in one go.Also take a look at Dynamodb-config-store: https://github.com/sebdah/dynamodb-config-store.
The Twelve-Factor App: http://12factor.net
How to manage production/staging/dev Django settings: https://discussion.heroku.com/t/how-to-manage-production-staging-dev-django-settings/21
An Introduction to boto’s DynamoDB interface: http://boto.readthedocs.org/en/2.3.0/dynamodb_tut.html