Calculate Price of EBS Volumes with Python

This post describes a tool that I have created for calculating monthly price for used and unused EBS volumes per EBS type. The reason for creating this tool is that AWS’ Cost Explorer did not provide necessary information for me to utilise it.

 

Background


My company has asked me to estimated monthly price of our application in one region, as we are about to deploy the application in another region, and estimated increase in costs was required. We are using mostly EC2 instances, some s3 buckets and lambda functions. The price of s3 storage is basically zero, since our application is not using a lot of it, and lambda functions are not utilised enough to even get a bill for them yet! The only thing left to calculate would be the EC2 instances, which is straight forward even though we use a variety of instance sizes. I found myself suck with EBS volumes, as we use a huge variety of sizes and EBS types for each instance. I quickly realised this will not be that easy.

 

I have made a Python script that would access AWS API to fetch EBS volumes per type and add them all up so that my calculation would be easier. I realised that I could push this further by querying Pricing API for EBS prices. After some failed tries querying the Pricing API, I have build a script that gives a tabular report of used and unused (unattached) EBS volumes and their monthly price.

 

 

Check it out here! Read the documentation and let me know how I can improve this project!

 

 

How to Fetch EBS Prices From Pricing API


During my research for this project I have realised that many people (including me) did not know how to get EBS prices from Pricing API. EBS volume prices are hidden under `AmazonEC2` pricing!

 

To get raw results from Pricing API use the following script. It takes region code as an argument e.g. eu-west-1

 

#!/usr/bin/env python3
import boto3
import pprint
import sys

aws_region_map = {
    'ca-central-1': 'Canada (Central)',
    'ap-northeast-3': 'Asia Pacific (Osaka-Local)',
    'us-east-1': 'US East (N. Virginia)',
    'ap-northeast-2': 'Asia Pacific (Seoul)',
    'us-gov-west-1': 'AWS GovCloud (US)',
    'us-east-2': 'US East (Ohio)',
    'ap-northeast-1': 'Asia Pacific (Tokyo)',
    'ap-south-1': 'Asia Pacific (Mumbai)',
    'ap-southeast-2': 'Asia Pacific (Sydney)',
    'ap-southeast-1': 'Asia Pacific (Singapore)',
    'sa-east-1': 'South America (Sao Paulo)',
    'us-west-2': 'US West (Oregon)',
    'eu-west-1': 'EU (Ireland)',
    'eu-west-3': 'EU (Paris)',
    'eu-west-2': 'EU (London)',
    'us-west-1': 'US West (N. California)',
    'eu-central-1': 'EU (Frankfurt)'
    }

ebs_name_map = {
    'standard': 'Magnetic',
    'gp2': 'General Purpose',
    'io1': 'Provisioned IOPS',
    'st1': 'Throughput Optimized HDD',
    'sc1': 'Cold HDD'
}

region = sys.argv[1]
resolved_region = aws_region_map[region]
aws_pricing_region = "us-east-1"
pricing_auth = boto3.client('pricing', region_name=aws_pricing_region)


for ebs_code in ebs_name_map:
    response = pricing_auth.get_products(ServiceCode='AmazonEC2', Filters=[
        {'Type': 'TERM_MATCH', 'Field': 'volumeType', 'Value': ebs_name_map[ebs_code]}, 
        {'Type': 'TERM_MATCH', 'Field': 'location', 'Value': resolved_region}])
    pprint.pprint(response)

 

The output is rather chaotic!

{'FormatVersion': 'aws_v1',
 'PriceList': ['{"product":{"productFamily":"Storage","attributes":{"storageMedia":"HDD-backed","maxThroughputvolume":"40 '
               '- 90 MB/sec","volumeType":"Magnetic","maxIopsvolume":"40 - '
               '200","servicecode":"AmazonEC2","usagetype":"EUC1-EBS:VolumeUsage","locationType":"AWS '
               'Region","location":"EU (Frankfurt)","servicename":"Amazon '
               'Elastic Compute Cloud","maxVolumeSize":"1 '
               'TiB","operation":"","maxIopsBurstPerformance":"Hundreds"},"sku":"WZCW2T5BKNTTAJG9"},"serviceCode":"AmazonEC2","terms":{"OnDemand":{"WZCW2T5BKNTTAJG9.JRTCKXETXF":{"priceDimensions":{"WZCW2T5BKNTTAJG9.JRTCKXETXF.6YS6EN2CT7":{"unit":"GB-Mo","endRange":"Inf","description":"$0.059 '
               'per GB-month of Magnetic provisioned storage - EU '
               '(Frankfurt)","appliesTo":[],"rateCode":"WZCW2T5BKNTTAJG9.JRTCKXETXF.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.0590000000"}}},"sku":"WZCW2T5BKNTTAJG9","effectiveDate":"2018-07-01T00:00:00Z","offerTermCode":"JRTCKXETXF","termAttributes":{}}}},"version":"20180719234605","publicationDate":"2018-07-19T23:46:05Z"}'],
 'ResponseMetadata': {'HTTPHeaders': {'content-length': '1166',
                                      'content-type': 'application/x-amz-json-1.1',
                                      'date': 'Fri, 20 Jul 2018 18:44:02 GMT',
                                      'x-amzn-requestid': 'df9a89a1-8c4c-11e8-92f9-c54735d924d8'},
                      'HTTPStatusCode': 200,
                      'RequestId': 'df9a89a1-8c4c-11e8-92f9-c54735d924d8',
                      'RetryAttempts': 0}}
{'FormatVersion': 'aws_v1',
 'PriceList': ['{"product":{"productFamily":"Storage","attributes":{"storageMedia":"SSD-backed","maxThroughputvolume":"160 '
               'MB/sec","volumeType":"General '
               'Purpose","maxIopsvolume":"10000","servicecode":"AmazonEC2","usagetype":"EUC1-EBS:VolumeUsage.gp2","locationType":"AWS '
               'Region","location":"EU (Frankfurt)","servicename":"Amazon '
               'Elastic Compute Cloud","maxVolumeSize":"16 '
               'TiB","operation":"","maxIopsBurstPerformance":"3000 for '
               'volumes <= 1 '
               'TiB"},"sku":"WHD78M63XVBREKT5"},"serviceCode":"AmazonEC2","terms":{"OnDemand":{"WHD78M63XVBREKT5.JRTCKXETXF":{"priceDimensions":{"WHD78M63XVBREKT5.JRTCKXETXF.6YS6EN2CT7":{"unit":"GB-Mo","endRange":"Inf","description":"$0.119 '
               'per GB-month of General Purpose SSD (gp2) provisioned storage '
               '- EU '
               '(Frankfurt)","appliesTo":[],"rateCode":"WHD78M63XVBREKT5.JRTCKXETXF.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.1190000000"}}},"sku":"WHD78M63XVBREKT5","effectiveDate":"2018-07-01T00:00:00Z","offerTermCode":"JRTCKXETXF","termAttributes":{}}}},"version":"20180719234605","publicationDate":"2018-07-19T23:46:05Z"}'],
 'ResponseMetadata': {'HTTPHeaders': {'content-length': '1205',
                                      'content-type': 'application/x-amz-json-1.1',
                                      'date': 'Fri, 20 Jul 2018 18:44:02 GMT',
                                      'x-amzn-requestid': 'dfbb7f22-8c4c-11e8-92f9-c54735d924d8'},
                      'HTTPStatusCode': 200,
                      'RequestId': 'dfbb7f22-8c4c-11e8-92f9-c54735d924d8',
                      'RetryAttempts': 0}}
{'FormatVersion': 'aws_v1',
 'PriceList': ['{"product":{"productFamily":"Storage","attributes":{"storageMedia":"SSD-backed","maxThroughputvolume":"320 '
               'MB/sec","volumeType":"Provisioned '
               'IOPS","maxIopsvolume":"20000","servicecode":"AmazonEC2","usagetype":"EUC1-EBS:VolumeUsage.piops","locationType":"AWS '
               'Region","location":"EU (Frankfurt)","servicename":"Amazon '
               'Elastic Compute Cloud","maxVolumeSize":"16 '
               'TiB","operation":""},"sku":"6EW9RF7RXHYJKH4W"},"serviceCode":"AmazonEC2","terms":{"OnDemand":{"6EW9RF7RXHYJKH4W.JRTCKXETXF":{"priceDimensions":{"6EW9RF7RXHYJKH4W.JRTCKXETXF.6YS6EN2CT7":{"unit":"GB-Mo","endRange":"Inf","description":"$0.149 '
               'per GB-month of Provisioned IOPS SSD (io1) provisioned storage '
               '- EU '
               '(Frankfurt)","appliesTo":[],"rateCode":"6EW9RF7RXHYJKH4W.JRTCKXETXF.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.1490000000"}}},"sku":"6EW9RF7RXHYJKH4W","effectiveDate":"2018-07-01T00:00:00Z","offerTermCode":"JRTCKXETXF","termAttributes":{}}}},"version":"20180719234605","publicationDate":"2018-07-19T23:46:05Z"}'],
 'ResponseMetadata': {'HTTPHeaders': {'content-length': '1151',
                                      'content-type': 'application/x-amz-json-1.1',
                                      'date': 'Fri, 20 Jul 2018 18:44:03 GMT',
                                      'x-amzn-requestid': 'dfda03a3-8c4c-11e8-92f9-c54735d924d8'},
                      'HTTPStatusCode': 200,
                      'RequestId': 'dfda03a3-8c4c-11e8-92f9-c54735d924d8',
                      'RetryAttempts': 0}}
{'FormatVersion': 'aws_v1',
 'PriceList': ['{"product":{"productFamily":"Storage","attributes":{"storageMedia":"HDD-backed","maxThroughputvolume":"500 '
               'MiB/s","volumeType":"Throughput Optimized '
               'HDD","maxIopsvolume":"500 - based on 1 MiB I/O '
               'size","servicecode":"AmazonEC2","usagetype":"EUC1-EBS:VolumeUsage.st1","locationType":"AWS '
               'Region","location":"EU (Frankfurt)","servicename":"Amazon '
               'Elastic Compute Cloud","maxVolumeSize":"16 '
               'TiB","operation":""},"sku":"MSGXHGPU2XUF77P6"},"serviceCode":"AmazonEC2","terms":{"OnDemand":{"MSGXHGPU2XUF77P6.JRTCKXETXF":{"priceDimensions":{"MSGXHGPU2XUF77P6.JRTCKXETXF.6YS6EN2CT7":{"unit":"GB-Mo","endRange":"Inf","description":"$0.054 '
               'per GB-month of Throughput Optimized HDD (st1) provisioned '
               'storage - EU (F '
               'rankfurt)","appliesTo":[],"rateCode":"MSGXHGPU2XUF77P6.JRTCKXETXF.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.0540000000"}}},"sku":"MSGXHGPU2XUF77P6","effectiveDate":"2018-07-01T00:00:00Z","offerTermCode":"JRTCKXETXF","termAttributes":{}}}},"version":"20180719234605","publicationDate":"2018-07-19T23:46:05Z"}'],
 'ResponseMetadata': {'HTTPHeaders': {'content-length': '1185',
                                      'content-type': 'application/x-amz-json-1.1',
                                      'date': 'Fri, 20 Jul 2018 18:44:03 GMT',
                                      'x-amzn-requestid': 'dff94b74-8c4c-11e8-92f9-c54735d924d8'},
                      'HTTPStatusCode': 200,
                      'RequestId': 'dff94b74-8c4c-11e8-92f9-c54735d924d8',
                      'RetryAttempts': 0}}
{'FormatVersion': 'aws_v1',
 'PriceList': ['{"product":{"productFamily":"Storage","attributes":{"storageMedia":"HDD-backed","maxThroughputvolume":"250 '
               'MiB/s","volumeType":"Cold HDD","maxIopsvolume":"250 - based on '
               '1 MiB I/O '
               'size","servicecode":"AmazonEC2","usagetype":"EUC1-EBS:VolumeUsage.sc1","locationType":"AWS '
               'Region","location":"EU (Frankfurt)","servicename":"Amazon '
               'Elastic Compute Cloud","maxVolumeSize":"16 '
               'TiB","operation":""},"sku":"ZYSKSVGZYKJ85CXE"},"serviceCode":"AmazonEC2","terms":{"OnDemand":{"ZYSKSVGZYKJ85CXE.JRTCKXETXF":{"priceDimensions":{"ZYSKSVGZYKJ85CXE.JRTCKXETXF.6YS6EN2CT7":{"unit":"GB-Mo","endRange":"Inf","description":"$0.03 '
               'per GB-month of Cold HDD (sc1) provisioned storage - EU '
               '(Frankfurt)","appliesTo":[],"rateCode":"ZYSKSVGZYKJ85CXE.JRTCKXETXF.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.0300000000"}}},"sku":"ZYSKSVGZYKJ85CXE","effectiveDate":"2018-07-01T00:00:00Z","offerTermCode":"JRTCKXETXF","termAttributes":{}}}},"version":"20180719234605","publicationDate":"2018-07-19T23:46:05Z"}'],
 'ResponseMetadata': {'HTTPHeaders': {'content-length': '1151',
                                      'content-type': 'application/x-amz-json-1.1',
                                      'date': 'Fri, 20 Jul 2018 18:44:03 GMT',
                                      'x-amzn-requestid': 'e0184525-8c4c-11e8-92f9-c54735d924d8'},
                      'HTTPStatusCode': 200,
                      'RequestId': 'e0184525-8c4c-11e8-92f9-c54735d924d8',
                      'RetryAttempts': 0}}

 

The following Python script queries Pricing API and extracts prices from the chaotic JSON!

#!/usr/bin/env python3

import boto3
import pprint
import json
import sys

aws_region_map = {
    'ca-central-1': 'Canada (Central)',
    'ap-northeast-3': 'Asia Pacific (Osaka-Local)',
    'us-east-1': 'US East (N. Virginia)',
    'ap-northeast-2': 'Asia Pacific (Seoul)',
    'us-gov-west-1': 'AWS GovCloud (US)',
    'us-east-2': 'US East (Ohio)',
    'ap-northeast-1': 'Asia Pacific (Tokyo)',
    'ap-south-1': 'Asia Pacific (Mumbai)',
    'ap-southeast-2': 'Asia Pacific (Sydney)',
    'ap-southeast-1': 'Asia Pacific (Singapore)',
    'sa-east-1': 'South America (Sao Paulo)',
    'us-west-2': 'US West (Oregon)',
    'eu-west-1': 'EU (Ireland)',
    'eu-west-3': 'EU (Paris)',
    'eu-west-2': 'EU (London)',
    'us-west-1': 'US West (N. California)',
    'eu-central-1': 'EU (Frankfurt)'
    }

ebs_name_map = {
    'standard': 'Magnetic',
    'gp2': 'General Purpose',
    'io1': 'Provisioned IOPS',
    'st1': 'Throughput Optimized HDD',
    'sc1': 'Cold HDD'
}

region = sys.argv[1]
resolved_region = aws_region_map[region]
aws_pricing_region = "us-east-1"
pricing_auth = boto3.client('pricing', region_name=aws_pricing_region)


for ebs_code in ebs_name_map:
    response = pricing_auth.get_products(ServiceCode='AmazonEC2', Filters=[
        {'Type': 'TERM_MATCH', 'Field': 'volumeType', 'Value': ebs_name_map[ebs_code]}, 
        {'Type': 'TERM_MATCH', 'Field': 'location', 'Value': resolved_region}])
    for result in response['PriceList']:
        json_result = json.loads(result)
        for json_result_level_1 in json_result['terms']['OnDemand'].values():
            for json_result_level_2 in json_result_level_1['priceDimensions'].values():
                for price_value in json_result_level_2['pricePerUnit'].values():
                    continue
    ebs_name_map[ebs_code] = float(price_value)

pprint.pprint(ebs_name_map, width=5)

 

Output is what you can work with in your projects!

{
 'gp2': 0.119,
 'io1': 0.149,
 'sc1': 0.03,
 'st1': 0.054,
 'standard': 0.059
}
Katapult Cloud
No Comments

Post a Comment

Comment
Name
Email
Website