Make it durable¶

Use Object Storage to store fractals¶

The Fractals app currently uses the local file system on the instance to store the images that it generates. For a number of reasons, this approach is not scalable or durable.

Because the local file system is ephemeral storage, the fractal images are lost along with the instance when the instance is terminated. Block-based storage, which the Block Storage section discusses, avoids that problem, but like local file systems, it requires administration to ensure that it does not fill up, and immediate attention if disks fail.

The Object Storage service manages many of the tasks normally managed by the application owner. The Object Storage service provides a scalable and durable API that you can use for the fractals app, eliminating the need to be aware of the low level details of how objects are stored and replicated, and how to grow the storage pool. Object Storage handles replication for you. It stores multiple copies of each object. You can use the Object Storage API to return an object, on demand.

First, learn how to connect to the Object Storage endpoint:

from __future__ import print_function
from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver

auth_username = 'your_auth_username'
auth_password = 'your_auth_password'
auth_url = 'http://controller:5000'
project_name = 'your_project_name_or_id'
region_name = 'your_region_name'

provider = get_driver(Provider.OPENSTACK_SWIFT)
swift = provider(auth_username,
                 auth_password,
                 ex_force_auth_url=auth_url,
                 ex_force_auth_version='2.0_password',
                 ex_tenant_name=project_name,
                 ex_force_service_region=region_name)

To begin to store objects, we must first make a container. Call yours fractals:

container_name = 'fractals'
container = swift.create_container(container_name=container_name)
print(container)

You should see output such as:

<Container: name=fractals, provider=OpenStack Swift>

You should now be able to see this container appear in a listing of all containers in your account:

print(swift.list_containers())

You should see output such as:

[<Container: name=fractals, provider=OpenStack Swift>]

The next logical step is to upload an object. Find a photo of a goat online, name it goat.jpg, and upload it to your fractals container:

file_path = 'goat.jpg'
object_name = 'an amazing goat'
container = swift.get_container(container_name=container_name)
object = container.upload_object(file_path=file_path, object_name=object_name)

List objects in your fractals container to see if the upload was successful. Then, download the file to verify that the md5sum is the same:

objects = container.list_objects()
print(objects)

[<Object: name=an amazing goat, size=191874, hash=439884df9c1c15c59d2cf43008180048, provider=OpenStack Swift ...>]

object = swift.get_object(container_name, object_name)
print(object)

<Object: name=an amazing goat, size=954465, hash=7513986d3aeb22659079d1bf3dc2468b, provider=OpenStack Swift ...>

import hashlib
print(hashlib.md5(open('goat.jpg', 'rb').read()).hexdigest())

7513986d3aeb22659079d1bf3dc2468b

Finally, clean up by deleting the test object:

swift.delete_object(object)

Now, no more objects are available in the fractals container.

objects = container.list_objects()
print(objects)

[]

container_name = 'fractals'
container = swift.get_container(container_name)

import json

import requests

endpoint = 'http://IP_API_1'
params = { 'results_per_page': '-1' }
response = requests.get('%s/v1/fractal' % endpoint, params=params)
data = json.loads(response.text)
for fractal in data['objects']:
    response = requests.get('%s/fractal/%s' % (endpoint, fractal['uuid']), stream=True)
    container.upload_object_via_stream(response.iter_content(), object_name=fractal['uuid'])

for object in container.list_objects():
    print(object)

<Object: name=025fd8a0-6abe-4ffa-9686-bcbf853b71dc, size=61597, hash=b7a8a26e3c0ce9f80a1bf4f64792cd0c, provider=OpenStack Swift ...>
<Object: name=26ca9b38-25c8-4f1e-9e6a-a0132a7a2643, size=136298, hash=9f9b4cac16893854dd9e79dc682da0ff, provider=OpenStack Swift ...>
<Object: name=3f68c538-783e-42bc-8384-8396c8b0545d, size=27202, hash=e6ee0cd541578981c294cebc56bc4c35, provider=OpenStack Swift ...>

Extra features¶

for object in container.list_objects():
    container.delete_object(object)
swift.delete_container(container)

file_path = 'goat.jpg'
object_name = 'backup_goat.jpg'
extra = {'meta_data': {'description': 'a funny goat', 'created': '2015-06-02'}}
with open('goat.jpg', 'rb') as iterator:
    object = swift.upload_object_via_stream(iterator=iterator,
                                            container=container,
                                            object_name=object_name,
                                            extra=extra)

swift.ex_multipart_upload_object(file_path, container, object_name,
                                 chunk_size=33554432)

Next steps¶

You should now be fairly confident working with Object Storage. You can find more information about the Object Storage SDK calls at:

https://libcloud.readthedocs.org/en/latest/storage/api.html

Or, try one of these tutorial steps:

• Block Storage: Migrate the database to block storage, or use the database-as-a-service component.
• Orchestration: Automatically orchestrate your application.
• Networking: Learn about complex networking.
• Advice for developers new to operations: Get advice about operations.
• Going crazy: Learn some crazy things that you might not think to do ;)