Linux Scroll

Just another WordPress.com site

Monthly Archives: September 2011

Data Warehouse

From Wikipedia, the free encyclopedia

Data Warehouse Overview

In computing, a data warehouse (DW) is a database used for reporting and analysis. The data stored in the warehouse is uploaded from the operational systems. The data may pass through an operational data store for additional operations before it is used in the DW for reporting.

A data warehouse maintains its functions in three layers: staging, integration, and access.
Staging is used to store raw data for use by developers.
The integration layer is used to integrate data and to have a level of abstraction from users.
The access layer is for getting data out for users.

This definition of the data warehouse focuses on data storage. The main source of the data is cleaned, transformed, catalogued and made available for use by managers and other business professionals for data mining, online analytical processing, market research and decision support (Marakas & O’Brien 2009). However, the means to retrieve and analyze data, to extract, transform and load data, and to manage the data dictionary are also considered essential components of a data warehousing system. Many references to data warehousing use this broader context. Thus, an expanded definition for data warehousing includes business intelligence tools, tools to extract, transform and load data into the repository, and tools to manage and retrieve metadata.

Advertisements

ETL

From Wikipedia, the free encyclopedia

Extract, transform and load (ETL) is a process in database usage and especially in data warehousing that involves:

* Extracting data from outside sources
* Transforming it to fit operational needs (which can include quality levels)
* Loading it into the end target (database or data warehouse)

Real-life ETL cycle
The typical real-life ETL cycle consists of the following execution steps:

1. Cycle initiation
2. Build reference data
3. Extract (from sources)
4. Validate
5. Transform (clean, apply business rules, check for data integrity, create aggregates or disaggregates)
6. Stage (load into staging tables, if used)
7. Audit reports (for example, on compliance with business rules. Also, in case of failure, helps to diagnose/repair)
8. Publish (to target tables)
9. Archive
10. Clean up

Business intelligence

From Wikipedia, the free encyclopedia

Business intelligence (BI) mainly refers to computer-based techniques used in identifying, extracting,[clarification needed] and analyzing business data, such as sales revenue by products and/or departments, or by associated costs and incomes.

BI technologies provide historical, current and predictive views of business operations. Common functions of business intelligence technologies are reporting, online analytical processing, analytics, data mining, process mining, complex event processing, business performance management, benchmarking, text mining and predictive analytics.

Business intelligence aims to support better business decision-making. Thus a BI system can be called a decision support system (DSS). Though the term business intelligence is sometimes used as a synonym for competitive intelligence, because they both support decision making, BI uses technologies, processes, and applications to analyze mostly internal, structured data and business processes while competitive intelligence gathers, analyzes and disseminates information with a topical focus on company competitors. Business intelligence understood broadly can include the subset of competitive intelligence.

Business Intelligence can be applied to the following business purposes (MARCKM), in order to drive business value:

1. Measurement – program that creates a hierarchy of performance metrics (see also Metrics Reference Model) and Benchmarking that informs business leaders about progress towards business goals (AKA Business process management).
2. Analytics – program that builds quantitative processes for a business to arrive at optimal decisions and to perform Business Knowledge Discovery. Frequently involves: data mining, process mining, statistical analysis, Predictive analytics, Predictive modeling, Business process modeling, complex event processing.
3. Reporting/Enterprise Reporting – program that builds infrastructure for Strategic Reporting to serve the Strategic management of a business, NOT Operational Reporting. Frequently involves: Data visualization, Executive information system, OLAP
4. Collaboration/Collaboration platform – program that gets different areas (both inside and outside the business) to work together through Data sharing and Electronic Data Interchange.
5. Knowledge Management – program to make the company data driven through strategies and practices to identify, create, represent, distribute, and enable adoption of insights and experiences that are true business knowledge. Knowledge Management leads to Learning Management and Regulatory compliance/Compliance.

Increase solaris zone memory capping

Check zone resources:

# /bin/prctl -n zone.max-swap `pgrep -z appzone init`
process: 10371: /sbin/init
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
zone.max-swap
        privileged      44.8GB      -   deny                                 -
        system          16.0EB    max   deny                                 -
# /bin/prctl -n zone.max-locked-memory `pgrep -z appzone init`
process: 10371: /sbin/init
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
zone.max-locked-memory
        system          16.0EB    max   deny                                 -

To use memory caps on a server or inside a zone, the rcapadm utility needs to be run with the “-E” (enable memory caps) option to enable memory caps:

# rcapstat -z 1 1
rcapstat: rcapd is not active
# rcapadm
rcapadm: resource caps not configured
# rcapadm -E

Temporary increase zone.max-swap resource:

# /bin/prctl -n zone.max-swap -r -v 90G `pgrep -z appzone init`

Temporary increase zone.max-locked-memory resource:

# /bin/prctl -n zone.max-locked-memory -r -v 90G `pgrep -z appzone init`

Change the physical memory capping:

# rcapadm -z appzone -m 90G

Permanent setting:

# zonecfg -z appzone

    zonecfg:appzone> select capped-memory
    zonecfg:zone:capped-memory> set physical=90G
    zonecfg:zone:capped-memory> set swap=90G
    zonecfg:zone:capped-memory> set locked=90G
    zonecfg:zone:capped-memory> end
    zonecfg:zone:> commit

After zone reboot, the setting will make permanent.
Check zone setting:

# zonecfg -z appzone info

capped-memory:
        physical: 90G
        [swap: 90G]
        [locked: 90G]

Device write protected

Edit registry

cmd > Run > regedit
HKLM\System\CurrentControlSet\Control\StorageDevicePolicies

Add:
Reg_dword
WriteProtect = 0

Eject and plug-in device again.

AMD AHCI Registry

Copy in notepad, save as ahci.reg

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CriticalDeviceDatabase\pci#ven_1002&dev_4391] 
"Service"="ahcix86" 
"ClassGUID"="{4D36E96A-E325-11CE-BFC1-08002BE10318}"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CriticalDeviceDatabase\pci#ven_1002&dev_4393] 
"Service"="ahcix86" 
"ClassGUID"="{4D36E96A-E325-11CE-BFC1-08002BE10318}"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ahcix86] 
"Type"=dword:00000001 
"Start"=dword:00000000 
"Group"="SCSI miniport" 
"ErrorControl"=dword:00000001 
"ImagePath"="system32\\drivers\\ahcix86.sys" 
"tag"=dword:00000019 
"DisplayName"="ATI AMD AHCI Controller"

vmstat

vmstat – reports virtual memory statistics.
Page-ins – when pages are returned to physical memory, the event is called a page-in.
Page-outs – when the kernel detects that memory is running low, it attempts to free up memory by paging out.
when pages are written to disk, the event is called a page-out.

vmstat, delay 5 seconds, count 10

# vmstat 5 10
procs                      memory    swap          io     system cpu
r  b  w   swpd   free  buff cache  si  so   bi   bo   in    cs us  sy  id

free column = amount of free memory
si = page-ins
so = page-outs

HP-UX getconf

getconf – get system configuration values

Get HP-UX machine SERIAL NUMBER:

# getconf CS_MACHINE_SERIAL

Get HP-UX machine kernel bits:

# getconf KERNEL_BITS

Determine if system supports 32/64 bits:

# getconf HW_CPU_SUPP_BITS
# getconf CPU_CHIP_TYPE
# getconf CS_MACHINE_IDENT
# getconf MACHINE_IDENT
# getconf MACHINE_MODEL
# getconf MACHINE_SERIAL

Solaris install vmware tools

# mkdir /root

# cd /cdrom/vmwaretools

# cp vmware-solaris-tools.tar.gz /root/

# cd /root/

# ls

# gunzip -d vmware-solaris-tools.tar.gz

# ls -l

# tar -xvf vmware-solaris-tools.tar

# cd vmware-tools-distrib/

# ls

# ./vmware-install.pl

Check listening port

netstat – show network status

Use netstat:

# netstat -a | grep LISTEN

-a         Show  the  state  of  all  sockets,  all  routing  table
entries,  or  all interfaces, both physical and logical.
# netstat -anp |grep :80

-a        Show  the  state  of  all  sockets,  all  routing  table
entries,  or  all interfaces, both physical and logical.
-n       Show network  addresses  as  numbers.
-p       Show the net to media tables.

Use nmap:

# nmap Hostname