Group items tagged

f you are running a large Oracle data warehouse you should be using parallel

Like all tools you have to use parallel correctly; no more would we think of using a wrench to hammer a nail then should you think parallel is the answer to all performance problems. Sometimes parallel will make things worse, sometimes parallel will make performance less predictable.

Parallel introduces additional work to a query, simplistically we need to: split the query into multiple parallel processes, execute them, wait for the processes to complete and finally coordinate the results. This all takes time to do. Our time saving comes from being able to process multiple smaller chunks of data simultaneously. If the time to execute the step in parallel is not significantly faster than doing it without parallel then the additional overhead may make parallel processing a slower option; this is typically the case with small tables where a full tablescan or an indexed access is fast. Use too few parallel processes and we will not gain much in performance, too many and we risk starving the database of resource for other work or even slow our own process as it waits for resource. If you have implemented some form of CPU resource management on your system you may find that you experience delays as your parallel slaves ‘wait their turn’

Business intelligence is a data analysis process aimed at boosting business performance by helping corporate executives and other end users make more informed decisions.

Business intelligence (BI) is a technology-driven process for analyzing data and presenting actionable information to help corporate executives, business managers and other end users make more informed business decisions.

BI encompasses a variety of tools, applications and methodologies that enable organizations to collect data from internal systems and external sources, prepare it for analysis, develop and run queries against the data, and create reports, dashboards and data visualizations to make the analytical results available to corporate decision makers as well as operational workers.

Integration BI infrastructure: All tools in the platform use the same security, metadata, administration, portal integration, object model and query engine, and should share the same look and feel. Metadata management: Tools should leverage the same metadata, and the tools should provide a robust way to search, capture, store, reuse and publish metadata objects, such as dimensions, hierarchies, measures, performance metrics and report layout objects. Development tools: The platform should provide a set of programmatic and visual tools, coupled with a software developer's kit for creating analytic applications, integrating them into a business process, and/or embedding them in another application. Collaboration: Enables users to share and discuss information and analytic content, and/or to manage hierarchies and metrics via discussion threads, chat and annotations.

Information Delivery Reporting: Provides the ability to create formatted and interactive reports, with or without parameters, with highly scalable distribution and scheduling capabilities. Dashboards: Includes the ability to publish Web-based or mobile reports with intuitive interactive displays that indicate the state of a performance metric compared with a goal or target value. Increasingly, dashboards are used to disseminate real-time data from operational applications, or in conjunction with a complex-event processing engine. Ad hoc query: Enables users to ask their own questions of the data, without relying on IT to create a report. In particular, the tools must have a robust semantic layer to enable users to navigate available data sources. Microsoft Office integration: Sometimes, Microsoft Office (particularly Excel) acts as the reporting or analytics client. In these cases, it is vital that the tool provides integration with Microsoft Office, including support for document and presentation formats, formulas, data "refreshes" and pivot tables. Advanced integration includes cell locking and write-back. Search-based BI: Applies a search index to structured and unstructured data sources and maps them into a classification structure of dimensions and measures that users can easily navigate and explore using a search interface. Mobile BI: Enables organizations to deliver analytic content to mobile devices in a publishing and/or interactive mode, and takes advantage of the mobile client's location awareness.

Analysis Online analytical processing (OLAP): Enables users to analyze data with fast query and calculation performance, enabling a style of analysis known as "slicing and dicing." Users are able to navigate multidimensional drill paths. They also have the ability to write back values to a proprietary database for planning and "what if" modeling purposes. This capability could span a variety of data architectures (such as relational or multidimensional) and storage architectures (such as disk-based or in-memory). Interactive visualization: Gives users the ability to display numerous aspects of the data more efficiently by using interactive pictures and charts, instead of rows and columns. Predictive modeling and data mining: Enables organizations to classify categorical variables, and to estimate continuous variables using mathematical algorithms. Scorecards: These take the metrics displayed in a dashboard a step further by applying them to a strategy map that aligns key performance indicators (KPIs) with a strategic objective. Prescriptive modeling, simulation and optimization: Supports decision making by enabling organizations to select the correct value of a variable based on a set of constraints for deterministic processes, and by modeling outcomes for stochastic processes.

ERP vendor selection. As the following graph shows, the primary candidates for ERP software were SAP, Oracle, Microsoft, Epicor, and Infor:

The cloud question. Despite the hype, only 14 percent of respondents are using ERP delivered as Software as a Service (SaaS). Although the best cloud vendors can deliver superior security and reliability than most internal IT departments, market momentum to ERP in the cloud is not there yet, as the following diagram illustrates:

Important lessons. Implementing an ERP system is always complex because the deployment drives changes to both data and processes that extend across departmental boundaries inside the organization.

Schema on write This is the traditional approach for Business Intelligence. A model, often dimensional, is built as part of the design process. This model is an abstraction of the complexity of the underlying systems, put in business terms. The purpose of the model is to allow the business users to interrogate the data in a way they understand.

The model is instantiated through physical database tables and the date is loaded through an ETL (extract, transform and load) process that takes data from one or more source systems and transforms it to fit the model, then loads it into the model.

The key thing is that the model is determined before the data is finally written and the users are very much guided or driven by the model in how they query the data and what results they can get from the system. The designer must anticipate the queries and requests in advance of the user asking the questions.

What this means in practice is that when the BI Server component starts up, it creates and reserves a number of threads in advance, determined by a number of parameters including SERVER_THREAD_RANGE.

You can see these threads running and ready to perform tasks for the BI Server component by using a tool such as Process Explorer for Windows

Thinking it through a bit, any given single query is, to a certain extent, only really going to use a small part of the total amount of CPUs available on a server, because it’s not the BI Server that runs queries in parallel, it’s the underlying database. For example, a single analysis against a single Oracle Database datasource would only really need a single BI Server thread to handle the query request, but when the underlying database receives the query, it might use a large number of its CPUs to process the query, returning results back to the BI Server to then pass back to the Presentation Server for display to the user.

Any individual fish or pod of fish in your sea represents one lead.

Your Nemo will not be the first or the second fish that you catch. At the beginning, you will have very little information about the Nemo you would like to catch. You will start to examine your fish and create some criteria as to how Nemo should look like. In other words, you are qualifying your fish.

Lead = Any Fish in The Sea. Opportunity = Nemo

1. Project Sponsorship and Governance 2. Project Management 3. Development Team (Core Team) 4. Extended Project Team

1. Project Sponsorship and Governance IT and the business should form a BI steering committee to sponsor and govern design, development, deployment, and ongoing support. It needs both the CIO and a business executive, such as CFO, COO, or a senior VP of marketing/sales to commit budget, time, and resources. The business sponsor needs the project to succeed. The CIO is committed to what is being built and how.

2. Project Management Project management includes managing daily tasks, reporting status, and communicating to the extended project team, steering committee, and affected business users. The project management team needs extensive business knowledge, BI expertise, DW architecture background, and people management, project management, and communications skills. The project management team includes three functions or members: Project development manager - Responsible for deliverables, managing team resources, monitoring tasks, reporting status, and communications. Requires a hands-on IT manager with a background in iterative development. Must understand the changes caused by this approach and the impact on the business, project resources, schedule and the trade-offs. Business advisor - Works within the sponsoring business organization. Responsible for the deliverables of the business resources on the project's extended team. Serves as the business advocate on the project team and the project advocate within the business community. Often, the business advocate is a project co-manager who defers to the IT project manager the daily IT tasks but oversees the budget and business deliverables. BI/DW project advisor - Has enough expertise with architectures and technologies to guides the project team on their use. Ensures that architecture, data models, databases, ETL code, and BI tools are all being used effectively and conform to best practices and standards.

memory_limit - Recommended setting: 512M or higher. The memory_limit parameter mainly comes into play when executing large transactions such as mass update, export and import. If this setting is too low when trying to perform one of these actions, the end user will encounter a fatal error and the process will not complete. upload_max_filesize and post_max_size - Recommended setting: 30M or higher. Both of these settings work in conjunction with each other when uploading files through SugarCRM which includes future upgrades as well as document and note attachments. Please note that there is also a setting in the application which can limit file upload file size for end users so the settings in PHP should be high enough to allow any future upgrade files to be loaded without error. max_execution_time = Recommended setting: 300. This setting controls how long a PHP process will remain active. It is important to set this parameter to a value that will allow for large requests to complete if necessary but also will not hamper performance of the server if running too long.

In regards to PHP setup, the following parameters should be set with values as indicated:

Maximum upload size Admin > System Settings 30000000 (~30 MB) The 'Maximum upload size' controls the maximum file size your users can upload into Sugar. This setting should not exceed the post_max_size and upload_max_filesize parameters in your PHP configuration.

Save file to file system rather than into table

The balanced scorecard is one popular methodology for linking strategy, technology, and performance management. Other methodologies, such as applied information economics, combine statistical analysis, portfolio theory, and decision science in order to help firms calculate the economic value of better information. Whether you use a published methodology or develop your own approach in-house, the important point is to make sure your BI activities are keyed to generating real business value, not merely creating pretty, but useless, dashboards and reports.

Practic, avem de-a face cu coborârea din sferele abstracte a BI-ului tradiţional către „enterprise intelligence“; o formă de „democratizare“ a BI-ului a devenit accesibilă maselor de utilizatori finali, pe baza ideii că instrumentele specifice acestui concept (analiză, raportare, semnalare etc.) trebuie să permită şi să ofere suport pentru luarea deciziilor în timp real.

Potrivit specialiştilor, „mutaţia“ menţionată reprezintă o evoluţie naturală, realizată sub presiunea pieţei, care impune luarea tot mai rapidă a unor decizii din ce în ce mai complexe la nivelul managementului operaţional în mod cotidian. Este vorba, practic, de o reorientare a conceptului de BI, de la tradiţionalul „data-centric“ spre mai pragmaticul „process-centric“, menit să permită un răspuns mai agil la provocările din piaţă.

Astfel, aplicaţiile de BI operaţional nu mai sunt rezervate doar analiştilor de business din top management, ci sunt accesibile şi directorilor executivi, managerilor şi utilizatorilor finali cu putere decizională. Prin intermediul acestui nou concept, managerii departamentelor de vânzări şi staff-urile din centrele de suport beneficaiză de informaţii relaţionate cu lista de activităţi zilnice şi de workflow-uri şi ghiduri de analiză, care îi ajută să interpreteze şi să analizeze informaţiile pe baza cărora trebuie să ia deciziile.

OBIEE 11g skills, both in terms of new functionality (mapping, analyses, KPIs and Scorecards etc) and new infrastructure (WebLogic, EM, OPSS etc) A smattering of Essbase skills, focused mainly on the integration with OBIEE and Essbase (and the many workarounds and gotchas) Good ODI skills, both in terms of the basics, but also being able to write knowledge modules, integrate with OBIEE, deployment and migration Solid database skills – OBIEE gave the illusion through aggregates etc that database tuning was redundant, but time has shown it’s by far the biggest success factor in a project – get the database design and optimisation wrong, and your project is toast. You need to know partitioning, materialized views, index types, and increasingly, you need to get yourself on an Exadata project as customers are buying the technology but you can’t teach it to yourself at home BI Apps skills, but watch out for everything changing when BI Apps 11g comes out, and be prepared to learn the Fusion Apps and JDeveloper if you want to stay in the game Looking to the future, keep an eye on technologies such as in-memory (TimesTen), mid-tier caching (Coherence), plus technologies such as Business Activity Monitoring (BAM), “big data” (Hadoop, large data sets, NoSQL), complex event processing and maybe products such as Qlikview, just in case Oracle buys them, or at least to know what the competition are up to, or more importantly pitching to your boss

The other thing to bear in mind of course, if you’re an Oracle BI developer, is that you need to have great business, communication and data modeling skills.

a lot of articles calling data scientists and statisticians the jobs of the future

there are more immediate needs that, when addressed, will have a much greater business impact.

Right now we have huge opportunities to make the data more accessible, more “joinable” and more consumable. Leaders don’t want more data – they want more information they can use to run their businesses.

Hadoop is designed to efficiently process large volumes of information by connecting many commodity computers together to work in parallel. The theoretical 1000-CPU machine described earlier would cost a very large amount of money, far more than 1,000 single-CPU or 250 quad-core machines. Hadoop will tie these smaller and more reasonably priced machines together into a single cost-effective compute cluster.

All of which led me to FusionCharts, which is a brilliant set of flash charts and widgets.

All I had to do was figure out how to integrate it into my app. First I had to write a function to extract the data I needed from my database and output it as correctly-formatted XML. That bit was easy so I won't bore you with it.

Next I uploaded the Flash (SWF) file for my chart into my workspace. (Tell me something: when you upload an image to your application using Apex's image uploader you refer to it by pointing at # APP_IMAGES#, so how do you think you'd refer to a file you've uploaded using Apex's file uploader? #APP_FILES#? Wrong! Illogically, all files uploaded into your application should be pointed at using the #APP_IMAGES# substitution string.)Finally, I created a dynamic PL/SQL content region outputting the necessary wrapper tags for my Flash movie (which I copied from the FusionCharts examples), pointing it to my uploaded swf file and feeding it the XML from my database function (which I call in "before regions" page process).