partitioning a table question...

Hi,
I have to plan a datawarehouse structure where the primary reporting usage
is to list detailed information about customers.
And also, we have OLAP cube for analysis purpose.
I'll have 100 000 customers and a fact table containing between 35 to
45millions of rows by year. (we also have 2 other fact table with 5 to
6millions of rows by year)
Generally, this fact table is filtered for a particular organizational unit
and for data in 1 year. (for report generation using report server)
So, I plan to partition my fact table by organizational unit and by year.
But I have 180 units.
Does the partitioning will works fine with 180 * 5 years = 900 tables?
on this article:
http://msdn.microsoft.com/library/d...ndw.
htm
there is a note that the maximum number of tables is 256.
But the performance gain can be very high!
The estimated size of the DW is 10gb / year (maybe more with additional
indexes)
My DTS package is ready to support partioning table loading. (the package
automatically create the new table, indexes and update the view for each
detected partition in the staging source table)
My users access my reportserver interactively, scheduling anything is not an
option.
For higher analysis, my olap cubes are ready.
thanks for your feedback.
Jerome.Hi Jerome:
I have some kind of "rule". Just make a partition for each 1 GB of data.
Or doing it for a year, would be great. If you need more granularity, give
it a try to month partitioning.
Tell me if you need some help on this
Sincerely
Alejandro Leguizamo
MVP SQL Server
Colombia
"Jj" <willgart@.BBBhotmailAAA.com> wrote in message
news:uwuxtzxzEHA.3336@.TK2MSFTNGP11.phx.gbl...
> Hi,
> I have to plan a datawarehouse structure where the primary reporting usage
> is to list detailed information about customers.
> And also, we have OLAP cube for analysis purpose.
> I'll have 100 000 customers and a fact table containing between 35 to
> 45millions of rows by year. (we also have 2 other fact table with 5 to
> 6millions of rows by year)
> Generally, this fact table is filtered for a particular organizational
> unit and for data in 1 year. (for report generation using report server)
> So, I plan to partition my fact table by organizational unit and by year.
> But I have 180 units.
> Does the partitioning will works fine with 180 * 5 years = 900 tables?
> on this article:
> http://msdn.microsoft.com/library/d...nd
w.htm
> there is a note that the maximum number of tables is 256.
> But the performance gain can be very high!
> The estimated size of the DW is 10gb / year (maybe more with additional
> indexes)
> My DTS package is ready to support partioning table loading. (the package
> automatically create the new table, indexes and update the view for each
> detected partition in the staging source table)
> My users access my reportserver interactively, scheduling anything is not
> an option.
> For higher analysis, my olap cubes are ready.
> thanks for your feedback.
> Jerome.
>
>|||but does SQL Server support 500 tables in a partitioned view?
I don't need help about this, I just want to plan correctly.
and I don't want month partition but only year + organizational unit
Because I don't do sum or count aggregation in SQL (or just a little sum
compared of what my cube provide).
Generally my reports contains SQL statement like this:
* Last kown value at a specific date (subquery required) (by customer)
* events which start before and end after a specific date (between
statement) (by customer)
Its the reason of not using monthly partitions, because I never know when
these dates appear in the database.
"Alejo Leguizamo (MVP SQL)" <SQL@.sql.sql> a crit dans le message de news:
%23thX4VC0EHA.1204@.TK2MSFTNGP10.phx.gbl...
> Hi Jerome:
> I have some kind of "rule". Just make a partition for each 1 GB of data.
> Or doing it for a year, would be great. If you need more granularity, give
> it a try to month partitioning.
> Tell me if you need some help on this
> Sincerely
>
> --
> Alejandro Leguizamo
> MVP SQL Server
> Colombia
> "Jj" <willgart@.BBBhotmailAAA.com> wrote in message
> news:uwuxtzxzEHA.3336@.TK2MSFTNGP11.phx.gbl...
>|||Jerome,
Views are based on select statements. A select statement can have a max of
256 tables in the From clause. So the answer to your specific question is
no.
Danny
"Jj" <willgart@.BBBhotmailAAA.com> wrote in message
news:e1ghnJD0EHA.2624@.TK2MSFTNGP11.phx.gbl...
> but does SQL Server support 500 tables in a partitioned view?
> I don't need help about this, I just want to plan correctly.
> and I don't want month partition but only year + organizational unit
> Because I don't do sum or count aggregation in SQL (or just a little sum
> compared of what my cube provide).
> Generally my reports contains SQL statement like this:
> * Last kown value at a specific date (subquery required) (by customer)
> * events which start before and end after a specific date (between
> statement) (by customer)
> Its the reason of not using monthly partitions, because I never know when
> these dates appear in the database.
> "Alejo Leguizamo (MVP SQL)" <SQL@.sql.sql> a crit dans le message de news:
> %23thX4VC0EHA.1204@.TK2MSFTNGP10.phx.gbl...
>|||
sniff
I think I'll do this: partition by organization for the current year and by
year for the history... does this works in a partitioned view?
"Danny" <istdrs@.flash.net> a crit dans le message de news:
7Lkod.23022$Rf1.15232@.newssvr19.news.prodigy.com...
> Jerome,
> Views are based on select statements. A select statement can have a max
> of 256 tables in the From clause. So the answer to your specific question
> is no.
> Danny
> "Jj" <willgart@.BBBhotmailAAA.com> wrote in message
> news:e1ghnJD0EHA.2624@.TK2MSFTNGP11.phx.gbl...
>|||The primary key of the underlying tables in the view must be the same. For
this to work, you will have to query current and history separately with
each haing its own view.
Danny
"Jj" <willgart@.BBBhotmailAAA.com> wrote in message
news:uKoQZzP0EHA.1932@.TK2MSFTNGP09.phx.gbl...
>
> sniff
> I think I'll do this: partition by organization for the current year and
> by year for the history... does this works in a partitioned view?
> "Danny" <istdrs@.flash.net> a crit dans le message de news:
> 7Lkod.23022$Rf1.15232@.newssvr19.news.prodigy.com...
>

partitioning a table question...

Hi,
I have to plan a datawarehouse structure where the primary reporting usage
is to list detailed information about customers.
And also, we have OLAP cube for analysis purpose.
I'll have 100 000 customers and a fact table containing between 35 to
45millions of rows by year. (we also have 2 other fact table with 5 to
6millions of rows by year)
Generally, this fact table is filtered for a particular organizational unit
and for data in 1 year. (for report generation using report server)
So, I plan to partition my fact table by organizational unit and by year.
But I have 180 units.
Does the partitioning will works fine with 180 * 5 years = 900 tables?
on this article:
http://msdn.microsoft.com/library/de...itionsindw.htm
there is a note that the maximum number of tables is 256.
But the performance gain can be very high!
The estimated size of the DW is 10gb / year (maybe more with additional
indexes)
My DTS package is ready to support partioning table loading. (the package
automatically create the new table, indexes and update the view for each
detected partition in the staging source table)
My users access my reportserver interactively, scheduling anything is not an
option.
For higher analysis, my olap cubes are ready.
thanks for your feedback.
Jerome.
Hi Jerome:
I have some kind of "rule". Just make a partition for each 1 GB of data.
Or doing it for a year, would be great. If you need more granularity, give
it a try to month partitioning.
Tell me if you need some help on this
Sincerely
Alejandro Leguizamo
MVP SQL Server
Colombia
"Jj" <willgart@.BBBhotmailAAA.com> wrote in message
news:uwuxtzxzEHA.3336@.TK2MSFTNGP11.phx.gbl...
> Hi,
> I have to plan a datawarehouse structure where the primary reporting usage
> is to list detailed information about customers.
> And also, we have OLAP cube for analysis purpose.
> I'll have 100 000 customers and a fact table containing between 35 to
> 45millions of rows by year. (we also have 2 other fact table with 5 to
> 6millions of rows by year)
> Generally, this fact table is filtered for a particular organizational
> unit and for data in 1 year. (for report generation using report server)
> So, I plan to partition my fact table by organizational unit and by year.
> But I have 180 units.
> Does the partitioning will works fine with 180 * 5 years = 900 tables?
> on this article:
> http://msdn.microsoft.com/library/de...itionsindw.htm
> there is a note that the maximum number of tables is 256.
> But the performance gain can be very high!
> The estimated size of the DW is 10gb / year (maybe more with additional
> indexes)
> My DTS package is ready to support partioning table loading. (the package
> automatically create the new table, indexes and update the view for each
> detected partition in the staging source table)
> My users access my reportserver interactively, scheduling anything is not
> an option.
> For higher analysis, my olap cubes are ready.
> thanks for your feedback.
> Jerome.
>
>
|||but does SQL Server support 500 tables in a partitioned view?
I don't need help about this, I just want to plan correctly.
and I don't want month partition but only year + organizational unit
Because I don't do sum or count aggregation in SQL (or just a little sum
compared of what my cube provide).
Generally my reports contains SQL statement like this:
* Last kown value at a specific date (subquery required) (by customer)
* events which start before and end after a specific date (between
statement) (by customer)
Its the reason of not using monthly partitions, because I never know when
these dates appear in the database.
"Alejo Leguizamo (MVP SQL)" <SQL@.sql.sql> a crit dans le message de news:
%23thX4VC0EHA.1204@.TK2MSFTNGP10.phx.gbl...
> Hi Jerome:
> I have some kind of "rule". Just make a partition for each 1 GB of data.
> Or doing it for a year, would be great. If you need more granularity, give
> it a try to month partitioning.
> Tell me if you need some help on this
> Sincerely
>
> --
> Alejandro Leguizamo
> MVP SQL Server
> Colombia
> "Jj" <willgart@.BBBhotmailAAA.com> wrote in message
> news:uwuxtzxzEHA.3336@.TK2MSFTNGP11.phx.gbl...
>
|||Jerome,
Views are based on select statements. A select statement can have a max of
256 tables in the From clause. So the answer to your specific question is
no.
Danny
"Jj" <willgart@.BBBhotmailAAA.com> wrote in message
news:e1ghnJD0EHA.2624@.TK2MSFTNGP11.phx.gbl...
> but does SQL Server support 500 tables in a partitioned view?
> I don't need help about this, I just want to plan correctly.
> and I don't want month partition but only year + organizational unit
> Because I don't do sum or count aggregation in SQL (or just a little sum
> compared of what my cube provide).
> Generally my reports contains SQL statement like this:
> * Last kown value at a specific date (subquery required) (by customer)
> * events which start before and end after a specific date (between
> statement) (by customer)
> Its the reason of not using monthly partitions, because I never know when
> these dates appear in the database.
> "Alejo Leguizamo (MVP SQL)" <SQL@.sql.sql> a crit dans le message de news:
> %23thX4VC0EHA.1204@.TK2MSFTNGP10.phx.gbl...
>
|||
sniff
I think I'll do this: partition by organization for the current year and by
year for the history... does this works in a partitioned view?
"Danny" <istdrs@.flash.net> a crit dans le message de news:
7Lkod.23022$Rf1.15232@.newssvr19.news.prodigy.com.. .
> Jerome,
> Views are based on select statements. A select statement can have a max
> of 256 tables in the From clause. So the answer to your specific question
> is no.
> Danny
> "Jj" <willgart@.BBBhotmailAAA.com> wrote in message
> news:e1ghnJD0EHA.2624@.TK2MSFTNGP11.phx.gbl...
>
|||The primary key of the underlying tables in the view must be the same. For
this to work, you will have to query current and history separately with
each haing its own view.
Danny
"Jj" <willgart@.BBBhotmailAAA.com> wrote in message
news:uKoQZzP0EHA.1932@.TK2MSFTNGP09.phx.gbl...
>
> sniff
> I think I'll do this: partition by organization for the current year and
> by year for the history... does this works in a partitioned view?
> "Danny" <istdrs@.flash.net> a crit dans le message de news:
> 7Lkod.23022$Rf1.15232@.newssvr19.news.prodigy.com.. .
>

Parse field into multiple rows

Hello,
I am loading data from our MS Active Directory into our data
warehouse. (check out Mircosofts's Logparser, it can pull data from
ADS, server event logs and more. It can also create text files or load
directly to SQL. Its free and a pretty useful tool)
There is a field that contains the direct reports of a manager. The
direct report users are delimited by a pipe symbol.
I want to breakup the field into multple rows. There can be none, one
or many direct report users in this field.
<disclaimer>
This is a snippet of an example. This is only an example. I know that
I have not defined PK nor indexes. My focus is how to solve a problem
of parsing a field that has multple values into multple rows.
</disclaimer>
Thanks for any help in advance.
Rob

CREATE TABLE "dbo"."F_ADS_MANAGERS"
(
"MANAGER_KEY" VARCHAR(255) NULL,
"DIRECT_REPORTS_CN" VARCHAR(255) NULL
);

INSERT INTO F_ADS_MANAGERS (MANAGER_KEY, DIRECT_REPORTS_CN)
VALUES ('CN=Marilette, 'CN=Robert
D,OU=TechnologyGroup,DC=strayer,DC=edu|CN=Robert
Camarda,OU=TechnologyGroup,DC=strayer,DC=edu|CN=Mi chelle
C,OU=TechnologyGroup,DC=strayer,DC=edu|CN=Magnolia
B,OU=TechnologyGroup,DC=strayer,DC=edu|CN=Lee K,OU=TechnologyGroup')

I want to end up with 5 rows, 1 row for each user that is seprated by
the PIPE symbol.
CN=Marilette CN=Robert D,OU=TechnologyGroup,DC=strayer,DC=edu
CN=Marilette CN=Robert
Camarda,OU=TechnologyGroup,DC=strayer,DC=edu
CN=Marilette CN=Michelle C,OU=TechnologyGroup,DC=strayer,DC=edu
CN=Marilette CN=Magnolia B,OU=TechnologyGroup,DC=strayer,DC=edu
CN=Marilette CN=Lee K,OU=TechnologyGroupHere is one method to split the list using a recursive CTE (this assumes SQL
Server 2005). It is not the fastest, but with a small size list to split it
should be OK.

WITH Managers
(manager_key, direct_reports_cn, start_pos, end_pos)
AS
(
SELECT manager_key,
direct_reports_cn + '|',
1,
CHARINDEX('|', direct_reports_cn + '|')
FROM F_ADS_MANAGERS
UNION ALL
SELECT manager_key,
direct_reports_cn,
end_pos + 1,
CHARINDEX('|', direct_reports_cn, end_pos + 1)
FROM Managers
WHERE CHARINDEX('|', direct_reports_cn, end_pos + 1) 0
)
SELECT manager_key,
SUBSTRING(direct_reports_cn,
start_pos,
end_pos - start_pos) AS direct_report
FROM Managers
WHERE end_pos 0;

Erland Sommarskog has a very good article for arrays and lists that covers
various techniques and analysis:
http://www.sommarskog.se/arrays-in-sql.html
HTH,

Plamen Ratchev
http://www.SQLStudio.com|||Thanks for the link to Erland's article's, it is very handy.
I tried your suggestion but I am confused by the WITH.
I tried
SELECT * FROM F_ADS_MANAGERS
WITH Managers
<snip your code>
and I got an error. Haven't seen this before so I am not sure how to
trouble shoot.
Error I received:
Msg 336, Level 15, State 1, Line 2
Incorrect syntax near 'Managers'. If this is intended to be a common
table expression, you need to explicitly terminate the previous
statement with a semi-colon.

On Jun 27, 10:01 am, "Plamen Ratchev" <Pla...@.SQLStudio.comwrote:

Quote:

Originally Posted by

Here is one method to split the list using a recursive CTE (this assumes SQL
Server 2005). It is not the fastest, but with a small size list to split it
should be OK.
>
WITH Managers
(manager_key, direct_reports_cn, start_pos, end_pos)
AS
(
SELECT manager_key,
direct_reports_cn + '|',
1,
CHARINDEX('|', direct_reports_cn + '|')
FROM F_ADS_MANAGERS
UNION ALL
SELECT manager_key,
direct_reports_cn,
end_pos + 1,
CHARINDEX('|', direct_reports_cn, end_pos + 1)
FROM Managers
WHERE CHARINDEX('|', direct_reports_cn, end_pos + 1) 0
)
SELECT manager_key,
SUBSTRING(direct_reports_cn,
start_pos,
end_pos - start_pos) AS direct_report
FROM Managers
WHERE end_pos 0;
>
Erland Sommarskog has a very good article for arrays and lists that covers
various techniques and analysis:http://www.sommarskog.se/arrays-in-sql.html
>
HTH,
>
Plamen Ratchevhttp://www.SQLStudio.com

|||Here are a few notes that will help you understand the common table
expressions and why you get the error:

- It is required to terminate the statement before the WITH keyword defining
the common table expression with a semicolon. It is because the WITH keyword
has other uses. In your case if you end the select statement in the line
before WITH using ; it will work.
- I am not sure if you understand how CTEs work. CTEs are not materialized
and work pretty much like derived tables (with more functionality, like the
recursion used in this method). They only get defined before the statement
the uses them, so you can reference the CTE only in that statement. There is
a lot more to it, and if you decide to use this method I would suggest to
read more on CTEs in SQL Server Books OnLine.
- If you need to materialize the result of the CTE, you can use INSERT to
insert the data into a temporary table, or just define a view based on the
CTE (or you could also create a function based on the CTE).

Here is a complete example using the approach to define a view based on the
CTE:

CREATE TABLE F_ADS_MANAGERS
(
MANAGER_KEY VARCHAR(255) NULL,
DIRECT_REPORTS_CN VARCHAR(255) NULL
);

INSERT INTO F_ADS_MANAGERS (MANAGER_KEY, DIRECT_REPORTS_CN)
VALUES ('CN=Marilette',
'CN=Robert D,OU=TechnologyGroup,DC=strayer,DC=edu|
CN=Robert Camarda,OU=TechnologyGroup,DC=strayer,DC=edu|
CN=Michelle C,OU=TechnologyGroup,DC=strayer,DC=edu|
CN=Magnolia B,OU=TechnologyGroup,DC=strayer,DC=edu|
CN=Lee K,OU=TechnologyGroup');

GO

CREATE VIEW ManagersWithReports
(manager_key, direct_report)
AS
WITH Managers
(manager_key, direct_reports_cn, start_pos, end_pos)
AS
(
SELECT manager_key,
direct_reports_cn + '|',
1,
CHARINDEX('|', direct_reports_cn + '|')
FROM F_ADS_MANAGERS
UNION ALL
SELECT manager_key,
direct_reports_cn,
end_pos + 1,
CHARINDEX('|', direct_reports_cn, end_pos + 1)
FROM Managers
WHERE CHARINDEX('|', direct_reports_cn, end_pos + 1) 0
)
SELECT manager_key,
SUBSTRING(direct_reports_cn,
start_pos,
end_pos - start_pos) AS direct_report
FROM Managers
WHERE end_pos 0;

GO

SELECT manager_key,
direct_report
FROM ManagersWithReports;

GO

DROP VIEW ManagersWithReports;
DROP TABLE F_ADS_MANAGERS;

GO

HTH,

Plamen Ratchev
http://www.SQLStudio.com