Thursday, March 14, 2013

Where does SharePoint store list items?

In one of my previous posts, I've described where does SharePoint store all documents (like pdf, doc or xls files) from its document libraries.

But, items from other lists are not stored at the same place. They are stored in the same SQL database, but in a different table and in a very odd fashion.

NOTE: This post refers only to SharePoint 2010. 

NOTE 2: It needs to be noted that Microsoft does not recommend that you execute any SQL queries directly on SharePoint databases, or to be specific, it is not allowed.

But, sometimes, it is the easiest and quickest way of getting data you need.  

List of all SharePoint lists is located in dbo.AllLists table in WSS_Content database. Next image shows query in SQL Management Studio: 



Name of your list can be found in tp_Title column. If you want to see all the data of that list, then you need to copy its ID from tp_ID column and go to dbo.AllUserData table. ID of that list can also be found in the url when you open list settings in SharePoint:



Server_Name/Site_Name/_layouts/ListEdit.aspx?List={4B656BB9-0193-49D9-B5E4-0D76B6601198}


Now, if you go to dbo.AllUserData table with following query, you can see data of you SharePoint list:




Now, the columns which you get with this query have name like: nvarchar1, nvarchar2…int1, int2…datetime1…float1. 

If you want to know in which column is your data located, there are many ways, but I will explain three of them.

First, and very stupid way is comparing data from SharePoint list with data in AllUserData table looking for the same name.

For example, name of SharePoint list is usually stored in nvarchar1 column, so, if you need to find your names of all your SharePoint lists, you can look for it with following SQL query:


SELECT     nvarchar1 AS Title
FROM       [WSS_Content].[dbo].[UserData]
WHERE     tp_ListId = '4B656BB9-0193-49D9-B5E4-0D76B6601198'



In AllUserData table, number of columns of certain type is fixed (for example: there are 8 columns of datatime type), but you can have more then 8 datatime columns in SharePoint list. If there are more then 8 columns of datatime type in SharePoint list (all 8 datatime columns ar filled), in SQL table AllUserData, this ninth data will go in the datatime1 column, but in the next row. So, now this list item will occupy two rows in SQL table, and this will be marked in tp_RowOrdinal column. Every new row will have the same tp_ID like others, but tp_RowOrdinal will be incremented by one with each new row, and each new row get 8 new datatime columns. Same procedure works with other types of data, but there does not have to be 8 columns of certain type, for example, there are 64 of nvarchar type columns.

In my next post, I will show you how can you see in which column is you data stored using Powershell and C#.

13 comments:

  1. It should be noted that Microsoft does not like when people get near the SQL databases.

    Touching (changing) ANY of the data is ENTIRELY UNSUPPORTED... attempts to call Microsoft support will result in "you touched it... not our problem"

    Repeated querying of the data can affect the performance, since the databases are HIGHLY tuned for the queries that MS wrote... and ONLY the queries that MS wrote... your use may impact usage analysis, which can impact auto tuning, and result in an impact to the standard usage... or your queries may simply impact the standard queries (table locks, lack of use of indexes, among other possibilities).

    Since the data is accessible via the UI and APIs, there should be no reason to touch the data directly.
    Ever.

    ReplyDelete
    Replies
    1. Then tell me, how you access the data for Reporting Services from a SharePoint Foundation with SQL Express installation (with Advanced Tools), wich doesn't allow you to integrate it directly with SharePoint?

      Delete
    2. @ScottBrickey, to be honest I find your comment to be quite naïve. The idea that a developer cannot handle a database is just plain dumb. The best applications have a well designed 3NF database with tables that actually represent the entities in the application. Case in point, MS Dynamics CRM has a great 3NF database where tables = entities. Fantastic design.

      Who cares whether MS will support database updates. Who needs their "help" anyway? And you comment about support is misleading. MS will support you if you pay them, just not under "warranty". And their license agreement basically says they don't have to support anything.

      There is nothing wrong with using SQL. In fact SharePoint has so many bugs and problems your point has absolutely no credibility.

      Of course you would use the API most of the time, but the developer should have the discretion of whether to use SQL if he so desires and understands the query or update well enough.

      I have seen SP powershell fail to update the database properly, leaving a corrupt database. There are situations where you may need to use SQL because of poor SP API design.

      Consider that an API is just an abstraction of SQL anyway. And some API's are badly designed, do not work on occasion and in some case do not have the desired methods!

      SO STOP SHOUTING YOUR IGNORANCE FOR ALL TO READ.

      Delete
    3. Mr. Anonymous #1 is spot on.

      Delete
  2. Hi,
    i have solution for this. You can use powershell scripts to get data from sharepoint list and insert into New sql database and then get data for reporting

    please read following links

    http://www.dotnetfunda.com/codes/show/6988/powershell-scripts-for-export-and-import-sharepoint-list-to-sql-table


    https://www.linkedin.com/pulse/5-ways-export-sharepoint-data-sql-server-paul-gallagher

    ReplyDelete
  3. Hello!
    What do you think about deleting items from the list? Does rows in DB deleted too? Or they stay for centuries?

    ReplyDelete
    Replies
    1. There is a column in AllUserData table called tp_DeleteTransactionId. All items have this column set to "0x". "0x" indicates that item is not deleted (in SP list). When you delete list item, tp_DeleteTransactionId column of that row changes to some number (sth like 0x126553839289836526) and that means that item is deleted and it is now located in recycle bin. When you restore that item from recycle bin back to list, its tp_DeleteTransactionId changes back to "0x".
      Now, if you delete list item from list and also from recycle bin, then the row is also deleted from database.
      I am not 100% sure about this, but I believe that this is how it works...

      Delete
  4. I am unable to see data of multi lookup column from this query. Can anyone help me out plz

    SELECT int1 AS Country
    FROM [WSS_Content].[dbo].[UserData]
    WHERE tp_ListId = '4B656BB9-0193-49D9-B5E4-0D76B6601198'

    ReplyDelete
  5. @haritha k
    '4B656BB9-0193-49D9-B5E4-0D76B6601198' is the Guid of my list, you need to find the Guid for your list. Go to SahrePoint and open settings of your list and you will find the Guid in the URL.

    ReplyDelete
  6. to delete data from here just cast the select statement into xml like
    delete from [Shar].[dbo].[AllUserData] where [tp_ListId]= 'F43FA06B-5C-4D28-8BC6-D738CB8618' and cast([tp_ColumnSet] as nvarchar(max)) like '%841A062C8A502F57852567B1006A07C2%'

    ReplyDelete
    Replies
    1. Doing 'select' on SP database is one thing, but you should be very careful with 'delete'.

      Delete
  7. This comment has been removed by a blog administrator.

    ReplyDelete