How to connect SharePoint with Amazon Athena Data through external lists

You can use the CData API Server to give your users the capabilities to access and update Amazon Athena data in SharePoint. This article shows how to create an external list of Amazon Athena data that is always up to date. You will use an external content type as a template to create the external list. The external content type enables connectivity through OData, a real-time data streaming protocol for mobile and other online applications. The API Server is an OData producer of Amazon Athena feeds.

After setting up the API Server, creating an external list with connectivity to Amazon Athena data consists of three basic steps:

1. Create the External Content Type
2. Import the External Content Type
3. Create the External List

This article also covers how to accomplish the following tasks:

• Limit Results
• Configure Pass-Through Authentication
• Configure Credentials Authentication

Getting Started

---

Set Up the API Server

If you have not already done so, download the CData API Server. Once you have installed the API Server, follow the steps below to begin producing secure Amazon Athena OData services:

Connect to Amazon Athena

To provide Amazon Athena data to SharePoint users as an external list, we start by creating and configuring a Amazon Athena connection. Follow the steps below to configure the API Server to connect to Amazon Athena data:

1. First, navigate to the Connections page.
2. Click Add Connection and then search for and select the Amazon Athena connection. 👁 Selecting a data source (SQLite is shown)
   
3. Enter the necessary authentication properties to connect to Amazon Athena.

   Authenticating to Amazon Athena

   To authorize Amazon Athena requests, provide the credentials for an administrator account or for an IAM user with custom permissions: Set to the access key Id. Set to the secret access key.

   Note: Though you can connect as the AWS account administrator, it is recommended to use IAM user credentials to access AWS services.

   Obtaining the Access Key

   To obtain the credentials for an IAM user, follow the steps below:

   1. Sign into the IAM console.
   2. In the navigation pane, select Users.
   3. To create or manage the access keys for a user, select the user and then select the Security Credentials tab.

   To obtain the credentials for your AWS root account, follow the steps below:

   1. Sign into the AWS Management console with the credentials for your root account.
   2. Select your account name or number and select My Security Credentials in the menu that is displayed.
   3. Click Continue to Security Credentials and expand the Access Keys section to manage or create root account access keys.

   Authenticating from an EC2 Instance

   If you are using the CData Data Provider for Amazon Athena 2018 from an EC2 Instance and have an IAM Role assigned to the instance, you can use the IAM Role to authenticate. To do so, set to true and leave and empty. The CData Data Provider for Amazon Athena 2018 will automatically obtain your IAM Role credentials and authenticate with them.

   Authenticating as an AWS Role

   In many situations it may be preferable to use an IAM role for authentication instead of the direct security credentials of an AWS root user. An AWS role may be used instead by specifying the . This will cause the CData Data Provider for Amazon Athena 2018 to attempt to retrieve credentials for the specified role. If you are connecting to AWS (instead of already being connected such as on an EC2 instance), you must additionally specify the and of an IAM user to assume the role for. Roles may not be used when specifying the and of an AWS root user.

   Authenticating with MFA

   For users and roles that require Multi-factor Authentication, specify the and connection properties. This will cause the CData Data Provider for Amazon Athena 2018 to submit the MFA credentials in a request to retrieve temporary authentication credentials. Note that the duration of the temporary credentials may be controlled via the (default 3600 seconds).

   Connecting to Amazon Athena

   In addition to the and properties, specify , and . Set to the region where your Amazon Athena data is hosted. Set to a folder in S3 where you would like to store the results of queries.

   If is not set in the connection, the data provider connects to the default database set in Amazon Athena.

   👁 Connecting to a datasource (SQLite is shown)
   
4. After configuring the connection, click Save & Test to confirm a successful connection.

Configure API Server Users

Next, create a user to access your Amazon Athena data through the API Server. You can add and configure users on the Users page. Follow the steps below to configure and create a user:

1. On the Users page, click Add User to open the Add User dialog.
2. Next, set the Role, Username, and Privileges properties and then click Add User. 👁 Configure a new user
   
3. An Authtoken is then generated for the user. You can find the Authtoken and other information for each user on the Users page: 👁 API Server user settings
   

Creating API Endpoints for Amazon Athena

Having created a user, you are ready to create API endpoints for the Amazon Athena tables:

1. First, navigate to the API page and then click Add Table . 👁 Add tables
   
2. Select the connection you wish to access and click Next. 👁 Select the connection (SQLite is shown)
   
3. With the connection selected, create endpoints by selecting each table and then clicking Confirm. 👁 Adding tables from the connection (SQLite is shown)
   

Gather the OData Url

Having configured a connection to Amazon Athena data, created a user, and added resources to the API Server, you now have an easily accessible REST API based on the OData protocol for those resources. From the API page in API Server, you can view and copy the API Endpoints for the API:

👁 API Endpoints

Create the External Content Type

The external content type is a schema that will provide the core connectivity to Amazon Athena data from any SharePoint app. You can create a schema for any OData query. You can pass in the required options with the query string parameter. Below is an example request, which will return the schema in an .ect file:

https://my-server:8080/api.rsc/Customers?$sharepoint=AuthMode:Passthrough&@authtoken=my-authtoken

Note that for simplicity, the authtoken is passed in the query string to authenticate the request for the .ect. This is not enabled by default; if you would like to use this method to authenticate to the API Server, you will need to add an entry like the following to your settings.cfg file:

[Application]
AllowAuthtokenInUrl = true

The settings.cfg file is located in the data directory. In the .NET edition, the data directory is located in the app_data subfolder of the application root. In the Java edition, the location of the data directory depends on your operating system:

• Windows: C:\ProgramData\CData
• Unix or Mac OS X: ~/cdata

Import the External Content Type

After you have created the .ect, you can follow the steps below to import it into SharePoint Online or an on-premise SharePoint installation.

SharePoint 2013

Navigate to the SharePoint central administration portal and click the link to manage service applications. In the resulting page, click Business Data Connectivity Service. Select External Content Types in the menu and click Import. In the BDC Model section, click Choose File to select the .ect file in the dialog.

SharePoint Online

In the SharePoint admin center, click BCS from the quick launch menu and then click Manage BDC Models and External Content Types. On the resulting page, select External Content Types in the menu and click Import. In the BDC Model section, click Choose File.

Create the External List

You can now create SharePoint apps that can access and modify Amazon Athena data:

1. Navigate to your SharePoint site and choose Site Contents -> Add an App -> External List.
2. Click the Select External Content Type icon and choose the external content type that you created in the previous section.
3. Click Create.

👁 The external list. (Salesforce with SharePoint 2013 is shown.)

Limit Results

SharePoint has limits on how much data can be retrieved from external lists of OData sources. External lists display results in pages of 30 items by default. To modify the paging size, you can set the Item Limit property in the settings for the default view. Alternatively, you can build a custom Web part to view the data from the external list.

In SharePoint 2013, requests to external data sources are limited by bandwidth throttling controls, which can be changed using the Set-SPBusinessDataCatalogThrottleConfig command. In SharePoint Online, requests to external data sources are limited by your Server Resource Quota and by bandwidth throttling controls; to work around this, you can apply filters in the request for data.

The API Server sets the default limit for the number of returned rows to be 500. You can disable this limit by adding the limit option to the $sharepoint query string parameter and setting its value to 0.

Configure Pass-Through Authentication

To use pass-through authentication for accessing your external content type, set the AuthMode option to pass-through.

If you are using Kerberos authentication, you need to add these users to the API Server. If you are not using Kerberos authentication, you are likely using another form of Windows authentication such as NTLM.

When users are authenticating via NTLM authentication and using pass-through authentication in the Business Connectivity Service (BCS) in SharePoint, SharePoint connects to the external Web service using the default IIS account. This account is often the NT AUTHORITY\IUSR account. Give this user access to the CData API Server.

See the help documentation for a guide to enabling Windows authentication for the API Server.

Configure Credentials Authentication

In the following sections, you will first create a secure store target application that authenticates SharePoint users to the API Server with the credentials for a user who has been added to the API Server. Next, you will create the external content type and configure it to authenticate with the credentials in the secure store.

• SharePoint Online
• SharePoint 2013

SharePoint Online

1. In the SharePoint Online administration center, click secure store from the quick launch bar and then click New.
2. In the Target Application Settings section, enter the target application Id, display name, and contact email.
3. In the Credential Fields section, add the credentials for a user in the API Server.
4. In the Target Application Administrators section, choose an administrator who can access the connection settings for the external content type. The account for the SharePoint Online administrator is usually specified here.
5. In the Members section, enter the users in SharePoint who are authorized to access Amazon Athena data. 👁 The configuration page for the new target application in SharePoint Online. (Salesforce is shown.)
   

After you create the target application, save the credentials of The API Server user into the secure store:

1. In the quick launch bar, click secure store.
2. Click Set Credentials in the menu for the target application.
3. Enter the username and password of a user in the API Server.

Next, create a new connection settings object:

1. Click BCS from the quick launch bar and then choose the option to manage connections to online services.
2. Click Add. Name the connection and enter the URL of the OData endpoint, https://my-server/api.rsc.
3. Select the option to use credentials stored in SharePoint. Enter the application Id for the target application you created.

Finally, create the external content type, import it into SharePoint Online, and create the external list:

1. Generate the external content type by making a request for data and specifying the required options in the $sharepoint query string parameter. Set the AuthMode option to Credentials. Specify the target application Id and the Id of the connection settings object. For example:

   https://my-server/api.rsc/Customers?$filter=CustomerId eq '12345'&$sharepoint=AuthMode:Credentials,TargetApplicationId:my-target-application-Id,ODataConnectionSettingsId:my-odata-connection-settings-Id
   

2. Import the resulting .ect file: Open BCS from the quick launch bar and click the link to manage BDC models and external content types. Click Import.
3. You can now create the external list.

SharePoint 2013

1. Browse to the central administration area for your SharePoint site and click the link to manage service applications.
2. Click Secure Store Service and then click New.
3. Enter the application Id, display name, and contact email to configure the new secure store target application. Set the target application type to Group. 👁 Creating a new target application in SharePoint 2013. (Salesforce is shown.)
   
4. On the next page, add the field names and field types to display when the user enters their username and password. Select the username and password field types. 👁 The type of authentication used for the API Server.
   
5. Next, in the Target Application Administrators section, choose administrators who are authorized to access connection settings for the external content type. In the Members section, enter the users in SharePoint who are authorized to connect with the credentials of the API Server user. 👁 Administrators of the external content type and SharePoint users who are authorized to use the API Server.
   

After you create the new secure store target application, follow the procedure below to set the credentials that users in SharePoint will provide when accessing the API Server:

1. Click Set Credentials in the menu for the target application. 👁 The first step to save credentials to the API Server in the secure store. (Salesforce is shown.)
   
2. In the resulting menu enter the username and password (authtoken) of a user in the API Server. 👁 The credentials of a user in the API Server. (Salesforce is shown.)
   

3. Open an instance of the SharePoint management shell and initialize a new SharePoint connection object. Note that the connection object is site specific.

   New-SPODataConnectionSetting -AuthenticationMode Credentials -ServiceAddressURL http://my-server/api.rsc -ServiceContext http://myspsite/ -Name MyODataConnectionSettingsId -SecureStoreTargetApplicationId my-target-application-Id
    

   The output below shows that the command has executed successfully:

   👁 The initialized connection object.
   

The final steps involve creating the external content type for the table you want to expose in SharePoint, configuring it to authenticate with credentials from the secure store, and then importing it into SharePoint.

1. Generate the external content type by making a request for data and specifying the required options in the $sharepoint query string parameter: Below is an example request:

   http://my-server/api.rsc/Account?$filter=Industry eq 'Floppy Disks'&$sharepoint=AuthMode:Credentials,TargetApplicationId:my-target-application-Id,ODataConnectionSettingsId:my-odata-connection-settings-Id
   

2. To import the external content type into SharePoint, point your browser to the SharePoint central administration portal and click the link to manage service applications.
3. Click Business Data Connectivity Service in the resulting page.
4. Click Import and select the .ect file in the dialog.
5. You can now create the external list. On the Site Contents page on your SharePoint site, click the button to add an app and then click the icon to choose the external content type.