The issue you're experiencing with the "libexpat.so.1" error in AWS Lambda using Python 3.12 or 3.13 is likely related to changes in the Lambda execution environment for these newer Python versions.

Starting with Python 3.12, AWS Lambda uses Amazon Linux 2023 as the base operating system. This change in the underlying OS can lead to compatibility issues with certain libraries or dependencies that were present in previous Lambda environments.

To address this issue, you have a few options:

1. Use a custom runtime or container image: Since you're already using a Docker image, you can create a custom container image for your Lambda function. This allows you to include all necessary dependencies, including the libexpat library. Make sure your Dockerfile copies the required libraries and sets the appropriate environment variables.

2. Include the library in a Lambda layer: You can create a custom layer that includes the libexpat library and any other missing dependencies. Then, attach this layer to your Lambda function.

3. Modify your code to use a pure Python alternative: If possible, consider using a pure Python XML parsing library that doesn't rely on the system's libexpat.

To get more detailed error information:

1. Enable X-Ray tracing for your Lambda function to get more insights into the execution flow and potential issues.

2. Add more logging statements in your code, especially around the import statements, to pinpoint where exactly the error is occurring.

3. You can try to print out the contents of the /lib64 directory at the beginning of your Lambda function execution to verify if the libexpat.so.1 file is present in the runtime environment.

Regarding your Docker image, when you deploy a container image to Lambda, the entire container is used as the Lambda function's runtime environment. It's not transformed into a zip file or layer.

If you decide to stick with the container approach, ensure that your Dockerfile is set up correctly to include all necessary libraries and that the appropriate environment variables (like LD_LIBRARY_PATH) are set to point to the correct locations.

Remember to thoroughly test any changes in a non-production environment before deploying to production.
Sources
setlocale stopped working in lambda python 3.12 | AWS re:Post
Lambda execution environment has older version of GLIBC | AWS re:Post
snapstart for lambda container | AWS re:Post

Comment/reply:

Use a custom runtime or container image: Since you're already using a Docker image, you can create a custom container image for your Lambda function. This allows you to include all necessary dependencies, including the libexpat library. Make sure your Dockerfile copies the required libraries and sets the appropriate environment variables.

I did try to modify the CDK that I pointed at, and tried to swtch to aws_cdk.aws_lambda.Runtime.FROM_IMAGE. But I get the error message:

jsii.errors.JavaScriptError:
 ValidationError: runtime must be `Runtime.FROM_IMAGE` when using image asset for Lambda function
 at path [undefined]

I created a layer using docker based on Amazon Linux 23, because I want to create a Python 3.12 runtime compatible layer.

Here is a guide to create a lambda layer using docker: Set Up Your Project Folder:

Create a new folder on your computer specifically for building this layer. Let's call it geo-lambda-layer. Inside geo-lambda-layer, create a file named requirements.txt. Define Dependencies (requirements.txt):

Open the requirements.txt file you just created. List the Python packages your Lambda function needs, one per line. Based on your code, this should include:

requirements.txt

pandas geopandas

Note: geopandas will automatically pull in shapely, fiona, pyproj, numpy etc.

You can add specific versions if needed, e.g., geopandas==0.14.3

Save the file. Create the Dockerfile:

In the same geo-lambda-layer folder, create a file named Dockerfile (no file extension).

Open Dockerfile and paste the following content:

Dockerfile (Updated)

Use the official AWS Lambda Python 3.12 base image

Choose the base image matching your Lambda function's architecture (x86_64 or arm64)

For x86_64 (most common):

FROM public.ecr.aws/lambda/python:3.12-x86_64

For arm64 (AWS Graviton):

FROM public.ecr.aws/lambda/python:3.12-arm64

Create the directory structure Lambda expects for layers

RUN mkdir -p /opt/python/lib/python3.12/site-packages

--- NEW STEP: Install expat system library ---

Update package list and install expat using dnf

RUN echo "Installing system dependencies..." &&
dnf upgrade -y &&
dnf install -y expat findutils &&
echo "Copying required library files into layer..." &&
# Find and copy libexpat library files from system path to layer path # Common location on x86_64 AL2023 is /usr/lib64/. Adjust if needed for arm64. # Create the target lib directory if it doesn't exist mkdir -p /opt/python/lib/ &&
cp -L /usr/lib64/libexpat.so* /opt/python/lib/ &&
dnf clean all &&
echo "System dependencies processed and libraries copied."

Copy the requirements file into the container

COPY requirements.txt .

Update pip and install requirements into the target directory

Use the platform tag matching your chosen architecture

For x86_64: manylinux2014_x86_64

For arm64: manylinux2014_aarch64

RUN pip install --upgrade pip &&
pip install
--platform manylinux2014_x86_64
--target=/opt/python/lib/python3.12/site-packages
--implementation cp
--python-version 3.12
--only-binary=:all:
--requirement requirements.txt

--- NEW STEP: Reduce Layer Size ---

RUN echo "Starting size reduction..."

Remove pycache and *.pyc files

RUN find /opt/python/lib/python3.12/site-packages -name "pycache" -type d -exec rm -rf {} + RUN find /opt/python/lib/python3.12/site-packages -name "*.pyc" -type f -delete

Remove common test directories (adjust patterns if other names are used)

RUN find /opt/python/lib/python3.12/site-packages -type d -name "tests" -prune -exec rm -rf {} + RUN find /opt/python/lib/python3.12/site-packages -type d -name "test" -prune -exec rm -rf {} +

Remove distribution metadata (USE WITH CAUTION - can sometimes break pkg_resources/importlib.metadata)

Try commenting these out first if you encounter issues after building

RUN find /opt/python/lib/python3.12/site-packages -type d -name ".dist-info" -prune -exec rm -rf {} + RUN find /opt/python/lib/python3.12/site-packages -type d -name ".egg-info" -prune -exec rm -rf {} +

Remove some potentially large, non-essential data/docs within specific packages (Examples)

Adjust paths based on actual large directories you might find by exploring the built layer

RUN rm -rf /opt/python/lib/python3.12/site-packages/pandas/tests/data || echo "Pandas test data not found, skipping."

RUN rm -rf /opt/python/lib/python3.12/site-packages/geopandas/datasets || echo "GeoPandas datasets not found, skipping." # Usually small, but example

Optional: Strip symbols from shared object files (.so). Requires installing 'binutils'.

This can save significant space but might make debugging harder.

Uncomment the next line if needed, but try without it first.

RUN dnf install -y binutils && find /opt/python/lib/python3.12/site-packages -name "*.so" -type f -exec strip {} ; && dnf remove -y binutils && dnf clean all

RUN echo "Size reduction finished."

--- End Size Reduction ---

Optional: Clean up pip cache

RUN rm -rf /root/.cache/pip

--- end of Dockerfile

Then on terminal you have to do this: Go to this folder: geo-lambda-layer #build docker docker build -t geo-layer-builder .

#extract libraries from docker docker run --rm --entrypoint "" -v $(pwd)/layer-output:/output geo-layer-builder cp -r /opt/python /output/

go to libraries folder

cd layer-output

create the zip file with layer´s content

zip -r ../geo-lambda-layer.zip python

go to your AWS console

Upload to AWS Lambda:

Go to the AWS Lambda console. Navigate to "Layers" in the left-hand menu. Click "Create layer". Give your layer a name (e.g., GeopandasPython312). Upload the geo-lambda-layer.zip file you just created. Select the compatible architecture (x86_64 or arm64) and the compatible runtime (Python 3.12). Click "Create". Attach Layer to Function:

Go to your Lambda function's configuration page. Scroll down to the "Layers" section and click "Add a layer". Choose "Custom layers", select the layer you just created, choose the version, and click "Add".

Finally, I did this final setting in my lambda: Go to your Lambda function in the AWS Console. Go to Configuration > Environment variables. Click Edit. Add variable: Key: LD_LIBRARY_PATH Value: /opt/python/lib