It is not a common use-case, but occasionally we need to create a page or a document that contains the description of the Athena tables we have. It is relatively easy to do if we have written comments in the create external table statements while creating them because those comments can be retrieved using the boto3 client.
In this article, I am going to show you how to do it.
First, we have to create a glue client using the following statement:
import boto3
glue_client = boto3.client('glue',
region_name=region_name,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key)
To retrieve the tables, we need to know the database name:
glue_tables = glue_client.get_tables(DatabaseName=db_name, MaxResults=1000)
Want to build AI systems that actually work?
Download my expert-crafted GenAI Transformation Guide for Data Teams and discover how to properly measure AI performance, set up guardrails, and continuously improve your AI solutions like the pros.
Now, we can iterate over the tables and retrieve the data such as the column names, types, and the comments added when the table was created:
for table in glue_tables['TableList']:
for column in table['StorageDescriptor']['Columns']:
column_name = column['Name']
comment = column.get('Comment', '')
column_type = column['Type']
We have to remember that the code above does not return the columns used for data partitioning. To get the partition keys, we need the following code:
for table in glue_tables['TableList']:
for partition_key in table.get('PartitionKeys', []):
column_name = partition_key['Name']
comment = partition_key.get('Comment', '')
column_type = partition_key['Type']