How to connect SharePoint with CSV Data through external lists

You can use the CData API Server to give your users the capabilities to access CSV data in SharePoint. This article shows how to create an external list of CSV data that is always up to date. You will use an external content type as a template to create the external list. The external content type enables connectivity through OData, a real-time data streaming protocol for mobile and other online applications. The API Server is an OData producer of CSV feeds.

After setting up the API Server, creating an external list with connectivity to CSV data consists of three basic steps:

1. Create the External Content Type
2. Import the External Content Type
3. Create the External List

This article also covers how to accomplish the following tasks:

• Limit Results
• Configure Pass-Through Authentication
• Configure Credentials Authentication

Set Up the API Server

If you have not already done so, download the CData API Server. Once you have installed the API Server, follow the steps below to begin producing secure CSV OData services:

Connect to CSV

To provide CSV data to SharePoint users as an external list, we start by creating and configuring a CSV connection. Follow the steps below to configure the API Server to connect to CSV data:

1. First, navigate to the Connections page.
2. Click Add Connection and then search for and select the CSV connection. 👁 Selecting a data source (SQLite is shown)
   
3. Enter the necessary authentication properties to connect to CSV.

   Connecting to Local or Cloud-Stored (Box, Google Drive, Amazon S3, SharePoint) CSV Files

   CData Drivers let you work with CSV files stored locally and stored in cloud storage services like Box, Amazon S3, Google Drive, or SharePoint, right where they are.

   Setting connection properties for local files

   Set the URI property to local folder path.

   Setting connection properties for files stored in Amazon S3

   To connect to CSV file(s) within Amazon S3, set the URI property to the URI of the Bucket and Folder where the intended CSV files exist. In addition, at least set these properties:

   • AWSAccessKey: AWS Access Key (username)
   • AWSSecretKey: AWS Secret Key

   Setting connection properties for files stored in Box

   To connect to CSV file(s) within Box, set the URI property to the URI of the folder that includes the intended CSV file(s). Use the OAuth authentication method to connect to Box.

   Dropbox

   To connect to CSV file(s) within Dropbox, set the URI proprerty to the URI of the folder that includes the intended CSV file(s). Use the OAuth authentication method to connect to Dropbox. Either User Account or Service Account can be used to authenticate.

   SharePoint Online (SOAP)

   To connect to CSV file(s) within SharePoint with SOAP Schema, set the URI proprerty to the URI of the document library that includes the intended CSV file. Set User, Password, and StorageBaseURL.

   SharePoint Online REST

   To connect to CSV file(s) within SharePoint with REST Schema, set the URI proprerty to the URI of the document library that includes the intended CSV file. StorageBaseURL is optional. If not set, the driver will use the root drive. OAuth is used to authenticate.

   Google Drive

   To connect to CSV file(s) within Google Drive, set the URI property to the URI of the folder that includes the intended CSV file(s). Use the OAuth authentication method to connect and set InitiateOAuth to GETANDREFRESH.

   👁 Connecting to a datasource (SQLite is shown)
   
4. After configuring the connection, click Save & Test to confirm a successful connection.

Configure API Server Users

Next, create a user to access your CSV data through the API Server. You can add and configure users on the Users page. Follow the steps below to configure and create a user:

1. On the Users page, click Add User to open the Add User dialog.
2. Next, set the Role, Username, and Privileges properties and then click Add User. 👁 Configure a new user
   
3. An Authtoken is then generated for the user. You can find the Authtoken and other information for each user on the Users page: 👁 API Server user settings
   

Creating API Endpoints for CSV

Having created a user, you are ready to create API endpoints for the CSV tables:

1. First, navigate to the API page and then click Add Table . 👁 Add tables
   
2. Select the connection you wish to access and click Next. 👁 Select the connection (SQLite is shown)
   
3. With the connection selected, create endpoints by selecting each table and then clicking Confirm. 👁 Adding tables from the connection (SQLite is shown)
   

Gather the OData Url

Having configured a connection to CSV data, created a user, and added resources to the API Server, you now have an easily accessible REST API based on the OData protocol for those resources. From the API page in API Server, you can view and copy the API Endpoints for the API:

👁 API Endpoints

Create the External Content Type

The external content type is a schema that will provide the core connectivity to CSV data from any SharePoint app. You can create a schema for any OData query. You can pass in the required options with the query string parameter. Below is an example request, which will return the schema in an .ect file:

https://my-server:8080/api.rsc/Customer?$sharepoint=AuthMode:Passthrough&@authtoken=my-authtoken

Note that for simplicity, the authtoken is passed in the query string to authenticate the request for the .ect. This is not enabled by default; if you would like to use this method to authenticate to the API Server, you will need to add an entry like the following to your settings.cfg file:

[Application]
AllowAuthtokenInUrl = true

The settings.cfg file is located in the data directory. In the .NET edition, the data directory is located in the app_data subfolder of the application root. In the Java edition, the location of the data directory depends on your operating system:

• Windows: C:\ProgramData\CData
• Unix or Mac OS X: ~/cdata

Import the External Content Type

After you have created the .ect, you can follow the steps below to import it into SharePoint Online or an on-premise SharePoint installation.

SharePoint 2013

Navigate to the SharePoint central administration portal and click the link to manage service applications. In the resulting page, click Business Data Connectivity Service. Select External Content Types in the menu and click Import. In the BDC Model section, click Choose File to select the .ect file in the dialog.

SharePoint Online

In the SharePoint admin center, click BCS from the quick launch menu and then click Manage BDC Models and External Content Types. On the resulting page, select External Content Types in the menu and click Import. In the BDC Model section, click Choose File.

Create the External List

You can now create SharePoint apps that can access and modify CSV data:

1. Navigate to your SharePoint site and choose Site Contents -> Add an App -> External List.
2. Click the Select External Content Type icon and choose the external content type that you created in the previous section.
3. Click Create.

👁 The external list. (Salesforce with SharePoint 2013 is shown.)

Limit Results

SharePoint has limits on how much data can be retrieved from external lists of OData sources. External lists display results in pages of 30 items by default. To modify the paging size, you can set the Item Limit property in the settings for the default view. Alternatively, you can build a custom Web part to view the data from the external list.

In SharePoint 2013, requests to external data sources are limited by bandwidth throttling controls, which can be changed using the Set-SPBusinessDataCatalogThrottleConfig command. In SharePoint Online, requests to external data sources are limited by your Server Resource Quota and by bandwidth throttling controls; to work around this, you can apply filters in the request for data.

The API Server sets the default limit for the number of returned rows to be 500. You can disable this limit by adding the limit option to the $sharepoint query string parameter and setting its value to 0.

Configure Pass-Through Authentication

To use pass-through authentication for accessing your external content type, set the AuthMode option to pass-through.

If you are using Kerberos authentication, you need to add these users to the API Server. If you are not using Kerberos authentication, you are likely using another form of Windows authentication such as NTLM.

When users are authenticating via NTLM authentication and using pass-through authentication in the Business Connectivity Service (BCS) in SharePoint, SharePoint connects to the external Web service using the default IIS account. This account is often the NT AUTHORITY\IUSR account. Give this user access to the CData API Server.

See the help documentation for a guide to enabling Windows authentication for the API Server.

Configure Credentials Authentication

In the following sections, you will first create a secure store target application that authenticates SharePoint users to the API Server with the credentials for a user who has been added to the API Server. Next, you will create the external content type and configure it to authenticate with the credentials in the secure store.

• SharePoint Online
• SharePoint 2013

SharePoint Online

1. In the SharePoint Online administration center, click secure store from the quick launch bar and then click New.
2. In the Target Application Settings section, enter the target application Id, display name, and contact email.
3. In the Credential Fields section, add the credentials for a user in the API Server.
4. In the Target Application Administrators section, choose an administrator who can access the connection settings for the external content type. The account for the SharePoint Online administrator is usually specified here.
5. In the Members section, enter the users in SharePoint who are authorized to access CSV data. 👁 The configuration page for the new target application in SharePoint Online. (Salesforce is shown.)
   

After you create the target application, save the credentials of The API Server user into the secure store:

1. In the quick launch bar, click secure store.
2. Click Set Credentials in the menu for the target application.
3. Enter the username and password of a user in the API Server.

Next, create a new connection settings object:

1. Click BCS from the quick launch bar and then choose the option to manage connections to online services.
2. Click Add. Name the connection and enter the URL of the OData endpoint, https://my-server/api.rsc.
3. Select the option to use credentials stored in SharePoint. Enter the application Id for the target application you created.

Finally, create the external content type, import it into SharePoint Online, and create the external list:

1. Generate the external content type by making a request for data and specifying the required options in the $sharepoint query string parameter. Set the AuthMode option to Credentials. Specify the target application Id and the Id of the connection settings object. For example:

   https://my-server/api.rsc/Customer?$filter=FirstName eq 'Bob'&$sharepoint=AuthMode:Credentials,TargetApplicationId:my-target-application-Id,ODataConnectionSettingsId:my-odata-connection-settings-Id
   

2. Import the resulting .ect file: Open BCS from the quick launch bar and click the link to manage BDC models and external content types. Click Import.
3. You can now create the external list.

SharePoint 2013

1. Browse to the central administration area for your SharePoint site and click the link to manage service applications.
2. Click Secure Store Service and then click New.
3. Enter the application Id, display name, and contact email to configure the new secure store target application. Set the target application type to Group. 👁 Creating a new target application in SharePoint 2013. (Salesforce is shown.)
   
4. On the next page, add the field names and field types to display when the user enters their username and password. Select the username and password field types. 👁 The type of authentication used for the API Server.
   
5. Next, in the Target Application Administrators section, choose administrators who are authorized to access connection settings for the external content type. In the Members section, enter the users in SharePoint who are authorized to connect with the credentials of the API Server user. 👁 Administrators of the external content type and SharePoint users who are authorized to use the API Server.
   

After you create the new secure store target application, follow the procedure below to set the credentials that users in SharePoint will provide when accessing the API Server:

1. Click Set Credentials in the menu for the target application. 👁 The first step to save credentials to the API Server in the secure store. (Salesforce is shown.)
   
2. In the resulting menu enter the username and password (authtoken) of a user in the API Server. 👁 The credentials of a user in the API Server. (Salesforce is shown.)
   

3. Open an instance of the SharePoint management shell and initialize a new SharePoint connection object. Note that the connection object is site specific.

   New-SPODataConnectionSetting -AuthenticationMode Credentials -ServiceAddressURL http://my-server/api.rsc -ServiceContext http://myspsite/ -Name MyODataConnectionSettingsId -SecureStoreTargetApplicationId my-target-application-Id
    

   The output below shows that the command has executed successfully:

   👁 The initialized connection object.
   

The final steps involve creating the external content type for the table you want to expose in SharePoint, configuring it to authenticate with credentials from the secure store, and then importing it into SharePoint.

1. Generate the external content type by making a request for data and specifying the required options in the $sharepoint query string parameter: Below is an example request:

   http://my-server/api.rsc/Account?$filter=Industry eq 'Floppy Disks'&$sharepoint=AuthMode:Credentials,TargetApplicationId:my-target-application-Id,ODataConnectionSettingsId:my-odata-connection-settings-Id
   

2. To import the external content type into SharePoint, point your browser to the SharePoint central administration portal and click the link to manage service applications.
3. Click Business Data Connectivity Service in the resulting page.
4. Click Import and select the .ect file in the dialog.
5. You can now create the external list. On the Site Contents page on your SharePoint site, click the button to add an app and then click the icon to choose the external content type.