ATTRIBUTE</th>	VALUE</th></tr></thead>
c_version</td>	1.0.0</td></tr>
localtimestamp</td>	2021-10-03 13:58:00</td></tr>
sysdate</td>	2021-10-03 11:58:00</td></tr>
g_conf_check_sysdate</td>	2021-10-03 11:58:10</td></tr>
g_conf_exit_sysdate</td>	2021-10-03 12:58:00</td></tr>
g_conf_client_identifier</td>	{o,o} 11ECD3500002</td></tr>
g_conf_level</td>	5</td></tr>
level_name(g_conf_level)</td>	trace</td></tr>
g_conf_check_interval</td>	10</td></tr>
g_conf_enable_ascii_art</td>	true</td></tr>
g_conf_call_stack</td>	false</td></tr>
g_conf_user_env</td>	false</td></tr>
g_conf_apex_env</td>	false</td></tr>
g_conf_cgi_env</td>	false</td></tr>
g_conf_console_env</td>	false</td></tr>
g_counters.count</td>	1</td></tr>
g_timers.count</td>	2</td></tr>
g_saved_stack.count</td>	0</td></tr>
g_prev_error_msg</td>	</td></tr> </tbody></table> APEX Error Handling Function</h2> For APEX, Console comes with a so-called “Error Handling Function</a>” which can enter errors within the APEX runtime environment into the log table. If you want to use this, you have to enter this function in your application in the “Application Builder” under “Edit Application Properties > Error Handling > Error Handling Function”: `console.apex_error_handling</code>.</p>` APEX error handling with ASCII art enabled</figcaption> </figure> The Error Handling Function logs the technical error in the CONSOLE_LOGS table and writes a friendly message to the end user. It uses the APEX Text Message feature for the user-friendly messages in case of constraint violations, as described in this video</a> by Anton and Neelesh from Insum, which in turn is based on an idea by Roel Hartman in this blog post</a>. The APEX community rocks….</p> APEX Plug-in for capturing frontend errors</h2> Furthermore, there is an APEX Dynamic Action Plug-in, which enters JavaScript errors in the user’s browser via AJAX call into the log table. This way you also get to know if the frontend is not running smoothly for the users…</p> You have to make sure that Console is either installed in the parsing schema of the application or a synonym named CONSOLE has been created in the parsing schema which points to the package CONSOLE. Then you can install the plug-in in install/apex_plugin.sql</code> and create a Dynamic Action for the whole application on page zero:</p> Event: Page Load</li> Action: Oracle Instrumentation Console [Plug-in]</li> No further customization required (only a JavaScript file will be loaded)</li> </ul> For those interested in what the plug-in does: sources/apex_plugin_console.js</code>. This is currently a minimal implementation and may be improved in the future.</p> Sources of inspiration</h2> I am not alone in the world and without the others not much goes in software development. Here are a few links to sites or projects that inspired me on the way to my own logging tool:</p> JavaScript Console API</a></li> Logger</a></li> Instrumentation for PLSQL</a></li> PIT</a></li> Oracle Magazine - Sophisticated Call Stack Analysis</a></li> </ul> Special thanks to Dietmar Aust</a>, who gave me time and ideas with a discussion session at an early stage.</p> Project homepage</h2> Console is hosted on GitHub</a>.</p> Happy logging and debugging :-)</p> Ottmar</p> Create fast insert scripts 2021-01-31T00:00:00+00:00 A few weeks ago I released a new version of my PLEX project</a>. I improved the ability to export table data as insert scripts. Previously, PLEX was only able to export CSV data. The reason was that I originally developed PLEX to speed up the implementation of version control in Oracle DB projects, and the ability to export CSV data was intended to track changes in master data tables.</p> It turned out that a colleague of mine wanted PLEX to export insert scripts for all data in all tables for the initial deployment of a new application. I implemented an initial version, but was not happy with the runtime of the insert scripts. I now hear some people arguing that there are better tools than insert scripts for importing larger amounts of data - but sometimes this is the only accepted way because only SQLPlus is available to run the deployment scripts….</p> Then I stumbled across an older post by Connor McDonald on creating fast insert scripts</a>. I have integrated his performance hacks into PLEX.</p> You can use them as follows (see last parameter):</p> declare</span></span> l_zip_file blob;</span></span> begin</span></span> l_zip_file :</span>=</span> plex.to_zip(plex.backapp(</span></span> p_app_id </span>=></span> 100</span>, </span>-- parameter only available when APEX is installed</span></span> p_include_ords_modules </span>=></span> true, </span>-- parameter only available when ORDS is installed</span></span> p_include_object_ddl </span>=></span> true,</span></span> p_include_data </span>=></span> true,</span></span> p_data_as_of_minutes_ago </span>=></span> 0</span>, </span>-- Read consistent data with the resulting timestamp (SCN). Defaults to 0.</span></span> p_data_max_rows </span>=></span> 10000</span>, </span>-- Maximum number of rows per table. Defaults to 1000.</span></span> p_data_table_name_like </span>=></span> 'CAT\_%,USERS,ROLES,RIGHTS'</span>,</span></span> p_data_table_name_not_like </span>=></span> '%\_HIST,LOGGER%'</span>,</span></span> p_data_format </span>=></span> 'insert:10'</span>));</span></span> </span> -- Do something with the zip file.</span></span> -- Your code here...</span></span> end</span>;</span></span> /</span></span></code></pre> The p_data_format</code> parameter is new and can contain a comma-separated list of formats - currently the CSV and INSERT formats are supported. An example: csv,insert</code> exports for each table a CSV file and a SQL file with insert statements. For insert you can also specify the number of rows per insert all</code> statement (default value is 20) - example: csv,insert:10</code> or insert:5</code>.</p> As you can see from the p_data_table_name_not_like => '%\_HIST,LOGGER%'</code> parameter, PLEX also accepts lists with like filters. In our example %\_HIST,LOGGER%</code> is translated to:</p> where</span> ...</span></span> and</span> ( table_name </span>not like</span> '%\_HIST'</span> escape </span>'\'</span></span> and</span> table_name </span>not like</span> 'LOGGER%'</span> escape </span>'\'</span></span> and</span> ...</span></span> )</span></span></code></pre> For the parameter p_data_table_name_like => 'CAT\_%,USERS,ROLES,RIGHTS'</code> the tables are filtered as follows:</p> where</span> ...</span></span> and</span> ( table_name </span>like</span> 'CAT\_%'</span> escape </span>'\'</span></span> or</span> table_name </span>like</span> 'USERS'</span> escape </span>'\'</span></span> or</span> ...</span></span> )</span></span></code></pre> There is a good reason why the default p_data_max_rows</code> parameter is set to 1000</code> - if you don’t specify table name filters, all tables will be exported…</p> The resulting insert scripts look like this:</p> -- Script generated by PLEX version 2.4.2 - more infos here: https://github.com/ogobrecht/plex</span></span> -- Performance Hacks by Connor McDonald: https://connor-mcdonald.com/2019/05/17/hacking-together-faster-inserts/</span></span> -- For strange line end replacements a big thank to SQLPlus: https://support.oracle.com/epmos/faces/DocumentDisplay?id=2377701.1 (SQL Failed With ORA-1756 In Sqlplus But Works In SQL Developer)</span></span> prompt </span>-</span> insert</span> 847</span> rows into</span> USERS (exported </span>2021</span>-</span>01</span>-</span>31 14</span>:</span>24</span>:</span>30</span>.</span>703234000</span> +</span>00</span>:</span>00</span>)</span></span> set</span> define </span>off</span> feedback </span>off</span> sqlblanklines </span>on</span></span> alter session set</span> cursor_sharing </span>=</span> force</span>;</span></span> alter session set</span> nls_numeric_characters </span>=</span> '.,'</span>;</span></span> alter session set</span> nls_date_format </span>=</span> 'yyyy-mm-dd hh24:mi:ss'</span>;</span></span> alter session set</span> nls_timestamp_format </span>=</span> 'yyyy-mm-dd hh24:mi:ssxff'</span>;</span></span> alter session set</span> nls_timestamp_tz_format </span>=</span> 'yyyy-mm-dd hh24:mi:ssxff tzr'</span>;</span></span> begin</span></span> insert</span> all</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>1</span>,</span>'Ylcmsajbil'</span>,</span>'Fojkjryntnixzfh'</span>,</span>'qvspjgvwmtbi@ghovilkddx.mly'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>2</span>,</span>'Axify'</span>,</span>'Cofjlkwzxytdih'</span>,</span>'ajgttnqlds@minokpyfo.gu'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>3</span>,</span>'Ckyqqvuqrkuktb'</span>,</span>'Igacqwp'</span>,</span>'qpygabuhbrs@nsjxpgjlle.ze'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>4</span>,</span>'Noythhl'</span>,</span>'Gausfu'</span>,</span>'ngmgsbr@duyxqzn.hmyo'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>5</span>,</span>'Kpewbrecnfzsi'</span>,</span>'Nwbsnjh'</span>,</span>'xwlhcfaxko@uhqsibdojjp.hsm'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>6</span>,</span>'Qlzyizkl'</span>,</span>'Gwnaojlvyud'</span>,</span>'kzndqj@nsosenf.fm'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>7</span>,</span>'Xpepttcemrd'</span>,</span>'Ktaqqdnqyfvc'</span>,</span>'uhbnzezvz@buiptt.lkrm'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>8</span>,</span>'Ybyinr'</span>,</span>'Vngairocujhy'</span>,</span>'igvfzoegbh@hsepkqiwbst.evs'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>9</span>,</span>'Ctjixltj'</span>,</span>'Yvsiei'</span>,</span>'ozpspssyw@vooiyfuf.xeh'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>10</span>,</span>'Lttzwsavnozxu'</span>,</span>'Kcyjalvzrl'</span>,</span>'yvwowaqrpku@dyapdumb.fvi'</span>)</span></span> select</span> </span> from</span> dual;</span></span> </span> / S N I P /</span></span> </span> insert</span> all</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>841</span>,</span>'Gpkabykoveq'</span>,</span>'Gljhlrijqop'</span>,</span>'imnhrheyr@ypccyiu.ah'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>842</span>,</span>'Mwrfazphbvmekpw'</span>,</span>'Kxirzfth'</span>,</span>'fxoatt@frlbwbn.tf'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>843</span>,</span>'Hzxomvkliaxl'</span>,</span>'Mstdrrmgfmsy'</span>,</span>'gpeidglzwfa@hwyumsansy.fet'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>844</span>,</span>'Vwvwezfsd'</span>,</span>'Xtfouojiymtlu'</span>,</span>'zgsdtowsvt@ywfngnijgts.ozd'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>845</span>,</span>'Eljntjfkxx'</span>,</span>'Sifgii'</span>,</span>'gksggat@ubfmmdopqy.ly'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>846</span>,</span>'Ewvyjeudjb'</span>,</span>'Anihlpdgeg'</span>,</span>'gietuk@ezciwejuedy.nuf'</span>)</span></span> into</span> USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) </span>values</span> (</span>847</span>,</span>'Wscdamsgssmouf'</span>,</span>'Omtaofvlrjs'</span>,</span>'jrwyzftmbmo@gjylnuez.esq'</span>)</span></span> select</span> </span> from</span> dual;</span></span> end</span>;</span></span> /</span></span> commit</span>;</span></span> alter session set</span> cursor_sharing </span>=</span> exact;</span></span> set</span> define </span>on</span></span></code></pre> PLEX saves the NLS settings of the current session and makes sure that the insert scripts run with the same settings - this means that you should not get any number or date conversion errors. The performance hacks from Connor are alter session set cursor_sharing = force;</code> and insert all</code> - thanks for pointing that out Connor :-)</p> PLEX has many parameters to configure the export - the focus of PLEX is to quickly introduce version control to existing projects (see also this article about PLEX in general</a>). If you want to use PLEX only for exporting table data, then you should create a wrapper function for it that suits your personal needs - here is an example:</p> create or replace function</span> to_insert</span>(</span></span> p_table_name </span>varchar2</span>,</span></span> p_max_rows </span>integer default</span> 1000</span>,</span></span> p_batch_size </span>integer default</span> 20</span>)</span></span> return</span> clob</span></span> is</span></span> l_file_collection plex.tab_export_files;</span></span> begin</span></span> l_file_collection :</span>=</span> plex.backapp(</span></span> p_include_data </span>=></span> true,</span></span> p_data_max_rows </span>=></span> p_max_rows,</span></span> p_data_table_name_like </span>=></span> p_table_name,</span></span> p_data_format </span>=></span> 'insert:'</span> \|\|</span> to_char(p_batch_size),</span></span> --overwrite defaults to save work for plex:</span></span> p_include_templates </span>=></span> false,</span></span> p_include_runtime_log </span>=></span> false,</span></span> p_include_error_log </span>=></span> false);</span></span> return</span> l_file_collection(</span>1</span>).contents;</span></span> end</span>;</span></span> /</span></span></code></pre> Now you can always do a quick export like this: select to_insert('USERS', 10000) from dual;</code></p> Happy exporting and scripting :-)</p> Ottmar</p> New Version of Oracle Table API Generator Available 2020-12-26T00:00:00+00:00 Short before Christmas André and I released version 0.6.0 of our table API generator</a>. The last release was exactly two years ago - time flies. This blog post highlights some of the changes - the complete changelog is available on GitHub</a>.</p> Bulk Processing</h2> When you split your app over multiple schemas for higher data security you might need at some point the possibility to bulk process your data. Imagine the following, common schema setup (also see this nice video by Connor McDonald</a>, 20 minutes):</p> Data Schema: Holds your tables and provides table APIs for the other schemas, which have only select rights on the tables (or views) and execute rights on the table APIs</li> API Schema: Holds your business logic and provides business APIs to the app parsing schema</li> UI Schema: Has no own objects, only connect rights and is used as the parsing schema to run the application</li> </ul> This setup follows the principle of the least privilege. When you need to bulk process your data in the business logic (API schema), then you have a problem without bulk processing facilities in your table APIs. You need to implement business logic in the data schema or you grant update and/or delete rights to the API schema. The first solution is against the separation of concerns and the second one lowers your data security and maybe prevents you from not having triggers (more about this later in the audit columns section).</p> That is why we have now also bulk processing in the core of our generated table APIs. Here is how you can use it in your business logic:</p> /* EXAMPLE INSERT</span></span> To keep the memory usage low we break our inserts into</span></span> a bulk size of 1000 rows. You should adapt the bulk size</span></span> to your use case. /</span></span> declare</span></span> l_rows_tab data_schema.your_table_api.t_rows_tab;</span></span> l_number_bulks </span>integer</span> :</span>=</span> 100</span>;</span></span> l_bulk_size </span>integer</span> :</span>=</span> 1000</span>;</span></span> begin</span></span> l_rows_tab :</span>=</span> data_schema.your_table_api.t_rows_tab</span>()</span>;</span></span> l_rows_tab.extend(l_bulk_size);</span></span> </span> <<</span>number_bulks</span>>></span></span> for</span> z </span>in</span> 1</span> .. l_number_bulks </span>loop</span></span> </span> <<</span>bulk_size</span>>></span></span> for</span> i </span>in</span> 1</span> .. l_bulk_size </span>loop</span></span> l_rows_tab(i).column_1 :</span>=</span> your_logic_here;</span></span> l_rows_tab(i).column_2 :</span>=</span> your_logic_here;</span></span> end loop</span> bulk_size;</span></span> </span> data_schema.your_table_api.create_rows(l_rows_tab);</span></span> commit</span>;</span></span> </span> end loop</span> number_bulks;</span></span> end</span>;</span></span> /</span></span></code></pre>/ EXAMPLE UPDATE /</span></span> declare</span></span> l_rows_tab data_schema.your_table_api.t_rows_tab;</span></span> l_ref_cursor data_schema.your_table_api.t_strong_ref_cursor;</span></span> begin</span></span> -- optionally set bulk limit, default is 1000</span></span> -- data_schema.your_table_api.set_bulk_limit(500);</span></span> open</span> l_ref_cursor </span>for</span></span> select</span> </span> from</span> app_users </span>where</span> your_conditions_here;</span></span> </span> <<</span>outer_bulk</span>>></span></span> loop</span></span> l_rows_tab :</span>=</span> data_schema.your_table_api.read_rows(l_ref_cursor);</span></span> </span> <<</span>inner_data</span>>></span></span> for</span> i </span>in</span> 1</span> .. l_rows_tab.count</span></span> loop</span></span> --do your business logic here</span></span> l_rows_tab(i).column_1 :</span>=</span> your_logic_here;</span></span> end loop</span> inner_data;</span></span> </span> data_schema.your_table_api.update_rows(l_rows_tab);</span></span> commit</span>;</span></span> </span> exit </span>when</span> data_schema.your_table_api.bulk_is_complete;</span></span> end loop</span> outer_bulk;</span></span> </span> close</span> l_ref_cursor;</span></span> end</span>;</span></span> /</span></span></code></pre> The bulk deletion works the same way as the bulk update - the difference is that you do not modify your data before calling the delete_rows</code> method.</p> For more examples see the docs:</p> Basic usage</a></li> Performance tests</a></li> </ul> Support for Audit Columns</h2> Keeping the bulk processing in mind for performance reasons it is a bad idea to have triggers only to manage audit columns in a table. If you have your tables in an own schema and the business logic and the app cannot directly write to the tables then it is easy to prevent triggers. Therefore the table APIs need to support that and hides the audit columns for insert or update operations. This is what our generator now supports.</p> You can map your specific audit column names to the four often used names created</code>, created_by</code>, updated</code> and updated_by</code> by providing a mapping string to the new parameter p_audit_column_mappings</code> - an example:</p> -- TABLE DEFINITION --</span></span> create table</span> app_users</span> (</span></span> au_id </span>integer generated always as identity</span>,</span></span> au_first_name </span>varchar2</span>(</span>15</span> char</span>) ,</span></span> au_last_name </span>varchar2</span>(</span>15</span> char</span>) ,</span></span> au_email </span>varchar2</span>(</span>30</span> char</span>) </span>not null</span> ,</span></span> au_active_yn </span>varchar2</span>(</span>1</span> char</span>) </span>default</span> 'Y'</span> not null</span> ,</span></span> -- This is only for demo purposes. In reality we expect</span></span> -- more unified names and types for audit columns.</span></span> au_created_on </span>date not null</span> ,</span></span> au_created_by </span>char</span>(</span>15</span> char</span>) </span>not null</span> ,</span></span> au_updated_at </span>timestamp not null</span> ,</span></span> au_updated_by </span>varchar2</span>(</span>15</span> char</span>) </span>not null</span> ,</span></span> --</span></span> primary key</span> (au_id),</span></span> unique</span> (au_email),</span></span> check</span> (au_active_yn </span>in</span> (</span>'Y'</span>, </span>'N'</span>))</span></span> );</span></span> </span> begin</span></span> om_tapigen.compile_api(</span></span> p_table_name </span>=></span> 'APP_USERS'</span>,</span></span> p_audit_column_mappings </span>=></span> 'created=#PREFIX#_CREATED_ON, created_by=#PREFIX#_CREATED_BY, updated=#PREFIX#_UPDATED_AT, updated_by=#PREFIX#_UPDATED_BY'</span></span> );</span></span> end</span>;</span></span> /</span></span></code></pre> The #PREFIX#</code> in our example is optional (determined automatically by the generator) and replaced with the found column_prefix au</code>. Doing it this way you can provide always the same audit column mapping to all of your tables. I know some people hate column prefixes - but if you design your data models this way the generator can save you the work to align the mapping for each table.</p> Support for a Row Version Column</h2> The same reasons with triggers for the audit columns are valid for the support of a row version column. This can also be handled by our generator:</p> begin</span></span> om_tapigen.compile_api(</span></span> p_table_name </span>=></span> 'YOUR_TABLE'</span>,</span></span> p_row_version_column_mapping </span>=></span> '#PREFIX#_VERSION_ID=tag_global_version_sequence.nextval'</span></span> );</span></span> end</span>;</span></span> /</span></span></code></pre> You can also see here the before mentioned #PREFIX#</code> substitution. The difference is that you define here only your column and for the column the SQL expression which should be used for the row version column - in this case, a sequence.</p> What we cannot do here is to add always 1 to the previous value like it is possible in a trigger - the table API has no idea what the previous value was. So, we are a little bit limited to a SQL expression but the point is, you can let the generated API manage the row version column without having a trigger.</p> Tenant Support</h2> This is the third new column mapping definition. For all people who have not an enterprise version with the VPD (virtual private database) functionality you can define a tenant expression for one of your columns:</p> begin</span></span> om_tapigen.compile_api(</span></span> p_table_name </span>=></span> 'YOUR_TABLE'</span>,</span></span> p_tenant_column_mapping </span>=></span> '#PREFIX#_TENANT_ID=to_number(sys_context(''my_sec_ctx'',''my_tenant_id''))'</span></span> );</span></span> end</span>;</span></span> /</span></span></code></pre> For more information see the parameter docs for p_tenant_column_mapping</a>.</p> More</h2> A new optional 1:1 view can be generated (new parameter p_enable_one_to_one_view</code>)</li> All generated object names can now be customized (new parameters p_dml_view_name</code>, p_dml_view_trigger_name</code> and p_one_to_one_view_name</code>)</li> Double quoting of table and column names in the generated code can now be configured (new parameter p_double_quote_names</code>, default true)</li> A new helper function get_default_row</code> will be generated, when parameter p_enable_column_defaults</code> is set to true (to support the row based create_row function)</li> We have now unit tests driven by utPLSQL (it will be a permanent task to improve the tests with every new feature or bugfix)</li> </ul> Breaking Changes</h2> Where there is light, there is also shadow.</p> All the new features like bulk processing and audit column support make it hard, if not impossible, to generate a generic change log during the row processing. Also the check, if we have a modification and omit an update if not necessary is hard to implement and not good for the performance with the evolved generator. So we skipped the mentioned functionality.</p> As far as we know not many people use the generic changelog. If you do, you have to stick with the previous generator version or you have to find an alternative way to do it in the future. A good way seems to be the use of the Flashback Data Archive as it is performant and available in all database editions for free - and you can load existing data into the archive (see Tims article below):</p> Blog post by Connor McDonald: Triggers can provide auditing information, but there’s a future in flashback</a></li> Blog post by Tim Hall: Flashback Data Archive (FDA) Enhancements in Oracle Database 12c Release 1 (12.1)</a></li> Video by Connor McDonald: Flashback Data Archive</a> (14 minutes)</li> Video APEX Connect 2020 by Connor McDonald: Flashback – so much more</a> (47 minutes)</li> Video AskTOM Office Hours by Chris Saxon: The Incredible Flashback Data Archive</a> (53 minutes)</li> </ul> And the last Breaking change is, that you need at least Oracle DB 12.1 or higher. That should (hopefully) no problem for the most projects.</p> Conclusion</h2> A big THANK YOU to all contributors - the current version of the generator would not be possible without your feedback and help.</p> Be sure to test the new generator before migrating to it. We use it already in some projects and it should be pretty stable. Nevertheless, with so many changed things we cannot give you a guarantee for every edge case - by using it you accept the MIT license. Please check out the project page at GitHub</a> and open an issue</a> if you encounter a problem.</p> A happy new year and happy coding :-)</p> Ottmar</p> How to migrate Oracle Cloud wallet into existing auto_login_local wallet 2020-08-03T00:00:00+00:00 I started to store user credentials in wallet files for higher security in SQL scripts. If you do so you should also secure the access to your wallet files on the operating system level. This is especially true for auto-login wallets from the Oracle Cloud. To increase the security a bit further you can restrict the auto-login to the local user who created the wallet. To achieve this you need to create an own wallet with orapki</code> instead of mkstore</code>:</p> orapki</span> wallet create</span> \</span></span> -wallet "path/to/my/wallet"</span> \</span></span> -pwd myPassword</span> \</span></span> -auto_login_local</span></span></code></pre> You can then add new alias/user/password entries with mkstore (you need to add each alias to your tnsnames.ora):</p> mkstore</span> \</span></span> -wrl "path/to/my/wallet"</span> \</span></span> -createCredential myAlias myUser myPassword</span></span></code></pre> If you use only the instant client and cannot find mkstore and orapki, then have a look at my previous post</a> on how to simulate these tools.</p> The Migration</h2> I assume you have set an environment variable called TNS_ADMIN pointing to your TNS directory. In this directory, you have your wallet files and also your existing tnsnames.ora and sqlnet.ora to successfully connect to your local Oracle instances.</p> Now you need to extract the downloaded cloud wallet - for security reasons it makes sense to create a temporary subdirectory in your TNS directory and extract the cloud wallet there. You need to migrate the entries from tnsnames.ora</code> and sqlnet.ora</code> into your own TNS files and align the path to your wallet directory. Additionally, you should also copy over the file ojdbc.properties</code> if you do not have it already and you plan to use SQLcl or SQL Developer with your wallet.</p> My resulting sqlnet.ora:</p> WALLET_LOCATION = (SOURCE = (METHOD = file) (METHOD_DATA = (DIRECTORY = "C:\\path\\to\\my\\wallet")))</span></span> SSL_SERVER_DN_MATCH = yes</span></span> sqlnet.wallet_override = true</span></span></code></pre> My resulting tnsnames.ora:</p> myCloudDbName_high, myAlias = (description = (...</span></span> myCloudDbName_medium = (description = (...</span></span> myCloudDbName_low = (description = (...</span></span> myCloudDbName_tp = (description = (...</span></span> myCloudDbName_tpurgent = (description = (...</span></span> </span> myLocalDbName, myOtherAlias = (description = (...</span></span> ...</span></span></code></pre> That was the easy part, but how you migrate the needed private key and all certificates?</p> It took me some time to figure this out because most people seem to use the cloud wallet as it is. This is understandable since SQL Developer can direct use the zipped version of the wallet. But my target is not to manage the passwords in multiple places - the databases, multiple wallets, SQL Developer and possibly other tools. I want to manage the passwords only in the databases and in one wallet and use this wallet with all the tools - in my case SQLPlus, SQLcl and SQL Developer.</p> First I tried to export the private key and all certificates from the cloud wallet and then import it into my wallet. I was not successful with this mostly because of the private key and unknown alias names - maybe I was too tired from all the try and error things on how to achieve this. I came finally up with the idea to use the cloud wallet Java KeyStore files and convert them into my wallet with the orapki method jks_to_pkcs12</code>.</p> Migrate private key and certificate from cloud wallet:</p> orapki</span> wallet jks_to_pkcs12</span> \</span></span> -wallet "path/to/my/wallet"</span> \</span></span> -pwd myPassword</span> \</span></span> -keystore "path/to/cloud/wallet/keystore.jks"</span> \</span></span> -jkspwd myCloudWalletPassword</span></span></code></pre> Migrate trusted certificates from cloud wallet:</p> orapki</span> wallet jks_to_pkcs12</span> \</span></span> -wallet "path/to/my/wallet"</span> \</span></span> -pwd myPassword</span> \</span></span> -keystore "path/to/cloud/wallet/truststore.jks"</span> \</span></span> -jkspwd myCloudWalletPassword</span></span></code></pre> Finally, list the wallet content to see the migrated entries:</p> orapki</span> wallet display -wallet . -pwd myPassword</span></span> # or</span></span> orapki</span> wallet display -wallet . -pwd myPassword -complete</span></span></code></pre> The final file list in my TNS directory (without the Java KeyStore files from the cloud wallet):</p> cwallet.sso</span></span> cwallet.sso.lck</span></span> ewallet.p12</span></span> ewallet.p12.lck</span></span> ojdbc.properties</span></span> sqlnet.ora</span></span> tnsnames.ora</span></span></code></pre> I am now able to connect to my cloud database with the desired tools without providing user/password details (tested version):</p> SQLPlus (19.6): sqlplus /@myAlias</code></li> SQLcl (20.2): sql /@myAlias</code></li> SQL Developer (20.2): authentication type “OS”, connection type “Custom JDBC” with URL jdbc:oracle:thin:/@myAlias</code></li> </ul> If you forgot which aliases you have configured in your wallet:</p> mkstore</span> -wrl "path/to/my/wallet" -listCredential</span></span></code></pre> Hope this helps someone else.</p> Happy developing and scripting :-)</p> Ottmar</p> How to use mkstore and orapki with Oracle Instant Client 2020-07-29T00:00:00+00:00 For some years I use only the Oracle Instant Client</a>. I had too often trouble with the full client installation on my corporate PCs under Windows, especially in the old Windows 32/64 bit times. The instant client can be used by download and extract one basic zip file and adding an entry to your path environment variable. If you want to use SQLPlus and other tools you need to download two additional files - no drama, also without admin rights.</p> When I started to store the user details in wallet files I was wondering why Oracle does not deliver the mkstore and orapki tools with the instant client. I searched for a workaround and found some posts about it - including installing the full client or other software to get the tools. But I do not want to install additional software only for two small command-line tools. I found the final hint in an old post from Andriy Dmytrenko</a>. Seems that the two utilities are only shell scripts that are calling some Java libraries. The question was from which location I should take the needed libraries. Since I also have SQLcl</a> on my PC I found the libs there. Below my final shell scripts for Windows - put them somewhere and have the path pointing to the directory where they are stored. If you need them for Linux you will find a way to do it by yourself:</p> mkstore.bat:</p> @</span>echo off</span></span> rem</span> inspiration: https://andriydmytrenko.wordpress.com/2013/07/01/using-the-secure-external-password-store-with-instant-client/</span></span> </span> setlocal</span></span> </span> rem</span> get the command line arguments</span></span> set</span> args</span>=</span></span> :</span>loop</span></span> if</span> !%1==!</span> goto</span> :</span>done</span></span> set</span> args</span>=</span>%args% %</span>1</span></span> shift</span></span> goto</span> :</span>loop</span></span> :</span>done</span></span> </span> rem</span> set classpath for mkstore - align this to your local SQLcl installation</span></span> set</span> sqlcl</span>=</span>c:\path\to\sqlcl\lib</span></span> set</span> classpath</span>=</span>%sqlcl%\oraclepki.jar</span></span> set</span> classpath</span>=</span>%classpath%;%sqlcl%\osdt_core.jar</span></span> set</span> classpath</span>=</span>%classpath%;%sqlcl%\osdt_cert.jar</span></span> </span> rem</span> simulate mkstore command</span></span> java -classpath %classpath% oracle.security.pki.OracleSecretStoreTextUI %args%</span></span> </span> endlocal</span></span></code></pre> orapki.bat:</p> @</span>echo off</span></span> rem</span> inspiration: https://andriydmytrenko.wordpress.com/2013/07/01/using-the-secure-external-password-store-with-instant-client/</span></span> </span> setlocal</span></span> </span> rem</span> get the command line arguments</span></span> set</span> args</span>=</span></span> :</span>loop</span></span> if</span> !%1==!</span> goto</span> :</span>done</span></span> set</span> args</span>=</span>%args% %</span>1</span></span> shift</span></span> goto</span> :</span>loop</span></span> :</span>done</span></span> </span> rem</span> set classpath for orapki - align this to your local SQLcl installation</span></span> set</span> sqlcl</span>=</span>c:\path\to\sqlcl\lib</span></span> set</span> classpath</span>=</span>%sqlcl%\oraclepki.jar</span></span> set</span> classpath</span>=</span>%classpath%;%sqlcl%\osdt_core.jar</span></span> set</span> classpath</span>=</span>%classpath%;%sqlcl%\osdt_cert.jar</span></span> </span> rem</span> simulate orapki command</span></span> java -classpath %classpath% oracle.security.pki.textui.OraclePKITextUI %args%</span></span> </span> endlocal</span></span></code></pre> If you need a starter on using mkstore and orapki then Tim Hall from Oracle Base has covered you with an article about the secure external password store</a>.</p> Happy password storing :-)</p> Ottmar</p> Sample Data Sets for Oracle 2020-05-25T00:00:00+00:00 Sometimes you need only some small tables and some rows of data to play around with, to test things out. I think most of you immediately would say EMP, DEPT…</p> Sure, but a bit more data would be nice. Ok, OE and HR schema…</p> Sure, but I have no SYS rights on the database in my current project to install the schemas, can I have it in my current dev schema?…</p> Sure, but then I have to fiddle a bit with the scripts to make it run in my dev schema, I need it fast, no time at all…</p> Mmhhh, sounds this familiar to you? Maybe I can help…</p> The sample data sets from Oracle were normally created for a dedicated schema. I changed that and omitted all the schema creation parts of the scripts. Instead, all objects from a data set get a prefix - an example: All objects from the customer orders schema (one of the newer data sets from Oracle) are prefixed with CO_. This allows me to use all data sets in parallel in a single schema and easily identify the data set the object belongs to.</p> All table scripts are created with identity columns and the simplest possible options for easy readability - no fancy constraint and index names. Constraints and indexes will be renamed with global helper scripts after the table creation. Only real indexes are coded - foreign key indexes are generated with a global helper script too.</p> For each data set, you have always the same three scripts to handle the installation/data refresh/uninstallation.</p> List of Data Sets, Copyrights</h2> EMP & DEPT (2 tabs, 18 rows), source</a>, Copyright Oracle (MIT license</a>)</li> Order Entry & Human Resources (15 tabs, 3.002 rows), source oe</a>, source hr</a>, Copyright Oracle (MIT license</a>)</li> Customer Orders (5 tabs, 6.325 rows), source</a>, Copyright Oracle (MIT license</a>)</li> Sakila DVD Rental Store (16 tabs, 46.273 rows), source</a>, original source</a>, Copyright MySQL AB documentation team, DB Software Laboratory, Lukas Eder (BSD-3-Clause license</a>)</li> </ul> For more info about the Oracle sample schemas see the docs</a>.</p> List of Global Helper Scripts</h2> Disable/enable all foreign key constraints</li> Create missing foreign key indexes</li> Unify constraint names</li> Unify index names</li> Sync sequence values to data</li> </ul> You can find the project on GitHub</a>.</p> Hope this helps someone else…</p> Happy data modeling and testing :-)</p> Ottmar</p> P.S. For those of you who are working with Oracle APEX: Have a look under SQL Workshop > Utilities > Sample Datasets</p> Why I Like SQLPlus for Deployment Scripts 2020-01-02T00:00:00+00:00 The following simplified example deployment master script is given:</p> set</span> define </span>off</span> verify </span>off</span> feedback </span>off</span></span> whenever sqlerror exit sql.sqlcode</span> rollback</span></span> spool deployment.log</span></span> </span> prompt</span></span> prompt Example Deployment </span>Master</span> Script</span></span> prompt </span>========================================</span></span> </span> prompt Step </span>1</span>: Dummy </span>for</span> demonstration</span></span> -- normally we call here some other scripts like</span></span> -- @other-script.sql</span></span> </span> prompt Step </span>2</span>: Error test </span>in anonymous block</span></span> begin</span></span> raise_application_error(</span></span> -</span>20000</span>,</span></span> 'Error test for SQLPlus <> SQLcl comparison'</span></span> );</span></span> end</span>;</span></span> /</span></span> </span> prompt </span>========================================</span></span> prompt Deployment Done :</span>-</span>)</span></span> prompt</span></span> </span> spool </span>off</span></span></code></pre> When we call this in SQLPlus (version 19.5), we get the following output (in the console and also in deployment.log) with an indication of what went wrong with our deployment:</p> Example Deployment </span>Master</span> Script</span></span> ========================================</span></span> Step </span>1</span>: Dummy </span>for</span> demonstration</span></span> Step </span>2</span>: Error test </span>in anonymous block</span></span> begin</span></span> </span></span> ERROR </span>at line</span> 1</span>:</span></span> ORA</span>-</span>20000</span>: Error test </span>for SQL</span></span>Plus </span><></span> SQLcl comparison</span></span> ORA</span>-</span>06512</span>: </span>at line</span> 2</span></span></code></pre> Ok, this error message is stupid, but usually, I get some useful information here to fix my problem in the deployment script - especially when the problematic script is a bit longer…</p> When we call this in SQLcl (latest version 19.4 on Windows) we get the following output (in the console and also the deployment.log):</p> Example Deployment </span>Master</span> Script</span></span> ========================================</span></span> Step </span>1</span>: Dummy </span>for</span> demonstration</span></span> Step </span>2</span>: Error test </span>in anonymous block</span></span></code></pre> Mhhh… What was going on here? Why was my script terminating in step 2?</p> That’s one reason why I like SQLPlus a bit more than SQLcl for deployment scripts. The fast start time of SQLPlus is another reason. Nevertheless, SQLcl is a cool command-line tool.</p> Happy scripting :-)</p> Ottmar</p> PS: There is a small Twitter discussion</a> around this…</p> UPDATE January 3, 2020</h2> Jeff Smith has confirmed that this bug was fixed in SQLcl version 19.4</a>. I was using 19.4 - but on Windows. It turns out that the bug is still there - only on Windows, not Mac, not Linux…</p> So, if you are facing this problem on Windows then you have to wait for a future release or set as a workaround the feedback to on:</p> set</span> define </span>off</span> verify </span>off</span> feedback </span>on</span></span> <</span>snip</span>></span></span></code></pre> Here the output on my Windows 10 laptop:</p> Demo Deployment </span>Master</span> Script</span></span> ========================================</span></span> Step </span>1</span>: Dummy </span>for</span> demonstration</span></span> Step </span>2</span>: </span>Begin</span> error test </span>in anonymous block</span></span> Rollback</span></span> </span> Error starting </span>at line</span> : </span>10</span> File</span> @ C:\temp\deployment.sql</span></span> In</span> Command </span>-</span></span> begin</span></span> raise_application_error(</span>-</span>20000</span>, </span>'error test for SQLPlus <> SQLcl comparison'</span>);</span></span> end</span>;</span></span> Error report </span>-</span></span> ORA</span>-</span>20000</span>: error test </span>for SQL</span></span>Plus </span><></span> SQLcl comparison</span></span> ORA</span>-</span>06512</span>: </span>at line</span> 2</span></span> 20000</span>. </span>00000</span> -</span> "%s"</span></span> </span>Cause: The stored </span>procedure</span> 'raise_application_error'</span></span> was called which causes this error </span>to</span> be </span>generated</span>.</span></span> </span>Action</span>: Correct the problem </span>as</span> described </span>in</span> the error </span>message or</span> contact</span></span> the </span>application</span> administrator </span>or</span> DBA </span>for</span> more information.</span></span></code></pre> If you are on Mac OS or Linux then you have to update to the latest SQLcl version 19.4 or use also the described workaround.</p> UPDATE July 20, 2020</h2> The bug still exists in SQLcl 20.2 and SQL Developer 20.2 under Windows - we have to wait a bit longer…</p> UPDATE January 31, 2021</h2> For me it is working now with SQL Developer 20.4.0 under Windows and also with SQLcl 20.4.1, when I have the environment variable path pointing to the JDK delivered within SQL Developer (...\sqldeveloper\jdk\jre\bin\</code>). I had the path before pointing to another standalone Java 11 distribution and it was not working there…</p> Again: happy scripting :-)</p> Ottmar</p> Download BLOBs with SQLPlus 2020-01-01T00:00:00+00:00 Have you ever tried to spool/download BLOBS with SQLPlus? Some months ago I tried to find a way to download a collection of CLOBs or even better a zipped version of the collection as BLOB. I learned that BLOBs are not a valid datatype in SQLPlus when it comes to the spool command. I found a way by putting the CLOBs into a global temporary table, creating an intermediate SQL script with tons of select and spool commands, and finally spooling the CLOBs one by one by calling the intermediate SQL script. It was working, but far from elegant. Also, when I was in the home office with a less powerful connection, it was slowing down the whole export process significantly.</p> If you are using PLEX (PL/SQL Export Utilities) and this sounds familiar to you - the described method was the default for exporting an APEX app or your schema DDL with the delivered export script templates.</p> If you have no idea what I am talking about: PLEX is a PL/SQL package that is able to export an Oracle APEX app, ORDS modules, all schema objects and table data into a nice directory structure in one go ready to use for version control. Check out the project on GitHub</a> and this blog post on how to get started</a>.</p> But there is a better way: Simply convert the BLOB into a base64 encoded CLOB. I have no idea why this was not my first solution for the PLEX export templates. In the new version 2.1.0 of PLEX the templates are using this approach. Here is a simplified example.</p> File export.sql:</p> set</span> verify </span>off</span> feedback </span>off</span> heading </span>off</span></span> set</span> trimout </span>on</span> trimspool </span>on</span> pagesize </span>0</span> linesize </span>5000</span> long </span>100000000</span> longchunksize </span>32767</span></span> whenever sqlerror exit sql.sqlcode</span> rollback</span></span> variable contents clob</span></span> begin</span></span> :contents :</span>=</span> plex.to_base64(plex.to_zip(plex.backapp(</span></span> p_app_id </span>=></span> 100</span>,</span></span> p_include_ords_modules </span>=></span> true,</span></span> p_include_object_ddl </span>=></span> false,</span></span> p_include_data </span>=></span> false,</span></span> p_include_templates </span>=></span> true)));</span></span> end</span>;</span></span> /</span></span> set</span> termout </span>off</span></span> spool </span>"app_100.zip.base64"</span></span> print</span> contents</span></span> spool </span>off</span></span> set</span> termout </span>on</span></span></code></pre> Congratulations - You are the owner of a base64 encoded CLOB on your client disk. And now? Before you can unzip the file you need to decode it - depending on your operating system you can do this with your OS tools:</p> Windows: certutil -decode app_100.zip.base64 app_100.zip</code></li> Mac: base64 -D -i app_100.zip.base64 -o app_100.zip</code></li> Linux: base64 -d app_100.zip.base64 > app_100.zip</code></li> </ul> Here a simple Windows batch file, which calls the SQL script above, decodes the file and deletes the base64 version.</p> File export.bat</p> echo</span> exit </span>\|</span> sqlplus -S demo/oracle</span>@</span>localhost:1521/xepdb1 </span>@</span>export.sql</span></span> certutil</span> -decode app_100.zip.base64 app_100.zip</span></span> del</span> app_100.zip.base64</span></span></code></pre> The described method to download a BLOB is independent of the package PLEX - you only need a small helper function to encode the BLOB. I found an example provided by Tim Hall on his Oracle Base site</a>. Thanks for sharing Tim!</p> CREATE OR REPLACE FUNCTION</span> base64encode</span>(p_blob </span>IN</span> BLOB)</span></span> RETURN</span> CLOB</span></span> -- -----------------------------------------------------------------------------------</span></span> -- File Name : https://oracle-base.com/dba/miscellaneous/base64encode.sql</span></span> -- Author : Tim Hall</span></span> -- Description : Encodes a BLOB into a Base64 CLOB.</span></span> -- Last Modified: 09/11/2011</span></span> -- -----------------------------------------------------------------------------------</span></span> IS</span></span> l_clob CLOB;</span></span> l_step PLS_INTEGER :</span>=</span> 12000</span>; </span>-- make sure you set a multiple of 3 not higher than 24573</span></span> BEGIN</span></span> FOR</span> i </span>IN</span> 0</span> .. TRUNC((DBMS_LOB.getlength(p_blob) </span>-</span> 1</span> )</span>/</span>l_step) </span>LOOP</span></span> l_clob :</span>=</span> l_clob </span>\|\|</span> UTL_RAW.cast_to_varchar2(UTL_ENCODE.base64_encode(DBMS_LOB.substr(p_blob, l_step, i </span></span> l_step </span>+</span> 1</span>)));</span></span> END LOOP</span>;</span></span> RETURN</span> l_clob;</span></span> END</span>;</span></span> /</span></span></code></pre> This function is included in the newest version of PLEX (named to_base64) - you can download PLEX here</a>.</p> Happy scripting, exporting and version controlling :-)</p> Ottmar</p> Quick start - version control for existing Oracle projects 2019-07-20T00:00:00+00:00 Many Oracle projects still do not use version control. The reasons for this are manifold. Mostly it is assumed that the database is a safe place for the source code. With a working backup this is also correct, but in any case one loses the complete history of changes. Often in running projects there is not enough time to deal with the introduction of source code versioning, because at first sight there is no direct benefit to be seen. Who dares to take a second look under time pressure? This article wants to lower the hurdle for the introduction of a version management a little bit.</em></p> Note DOAG</h2> This article appeared in DOAG Red Stack magazine 03-2019 and is also available in the german original</a>. Unlike the original, updates may occur here on my blog if relevant things change. For example, since version two of the mentioned PLEX (PL/SQL Export Utilities) project, an APEX installation is no longer a requirement. This means that also pure Oracle database projects can use PLEX - read more in this blog entry</a>.</p> Introduction</h2> The first step is the most important one, as everyone knows who has ever finally set out after much hesitation - whatever it was about. The same is true for source code versioning. Once you’re on your way, you wonder how you got along for so long without it. A versioned source code base increases your comfort zone during development, and the time invested pays off several times over. But how do you take the first step as easily and quickly as possible? How to design the structure of a repository?</p> Workflow change: file-based working</h2> One of the most important questions is: Can I continue to work as before? The answer depends entirely on how one has worked so far. Basically, working with a source code versioning also means a change away from developing in the database. Changing and compiling code directly in the database or changing tables with the tool of one’s choice must evolve to file-based work. This means working with a file in the local source code repository with the tool of choice. The changes are then compiled from the file. For schema changes, scripts must contain the necessary DDL code. Depending on the tools used so far, this is rather a subtle difference when editing logic, but a big difference when changing database objects like tables. If a schema diff tool was previously used to transport changes from the development environment into production, then this might even look like a step backwards at first glance.</p> The point here is that you can only get a usable source code history if you are consistent about working file-based. There can also be an intermediate step towards this approach. One continues to work as before and regularly exports all objects of the schema to the source code management. This way one gets at least a certain ordered transparency of the changes. If then in the course of the project the advantages of this procedure become apparent, one usually still lacks the information, who executed the respective changes. In addition, the transparency gained quickly awakens desires such as “If I have the old code in the repository, then I may try something, I can go back if necessary”. The first hurdle is already taken and the openness to perhaps really get started is there.</p> That something like this can work even better is something that most developers come up with themselves. For example, creating a branch (development branch) in the local repository. This way you can try something out without running the risk of having to discard your changes due to an unexpected bug. You can easily switch to the main (parallel) branch, work on the bugfix, update your branch to the changed main branch and then just keep trying. All this without the danger of losing things or influencing other colleagues. Have I lost anyone now? No problem, just get started - only practice helps here.</p> Tool comparison: DDL export</h2> How do you accomplish the first step? How do you export the schema objects of your application into a well sorted directory structure? Most of the used graphical development tools provide an export function for this purpose. Depending on the tool, this is more or less suitable for building a source code repository. Therefore, let’s take a look at how the most commonly used tools SQL Developer, PL/SQL Developer and Toad can do this. A word beforehand: the intention of the schema export of these tools is actually to bring the objects into another schema, maybe even into another database, and not to create a source code repository. The following questions move us when it comes to building one:</p> Is one script file per object possible?</li> Are subdirectories per object type possible?</li> Are separate files for foreign keys possible? (Useful for simpler master scripts)</li> Can “Object already exist” errors be prevented? (Or in other words: are the scripts restartable?)</li> Can data be exported? (If possible in CSV format for tracking master data changes).</li> Is it possible to export an APEX app? (Possibly also broken down into the individual parts such as pages, shared components, etc.)</li> </ul> Figure 1 shows the result related to the question.</p> Criterion</th> SQL Dev.</th> PL/SQL Dev.</th> Toad</th></tr></thead> File per object</td> Yes</td> Yes</td> Yes</td></tr> Subdir. per type</td> Yes</td> No</td> Yes</td></tr> FK constr. separate</td> Yes</td> No</td> Yes</td></tr> Prevent “object exist”</td> No</td> No</td> No</td></tr> Export data</td> Yes</td> No</td> partly</strong></em></td></tr> Export APEX app</td> partly</strong></em></td> No</td> No</td></tr> </tbody></table> Figure 1: Tool comparison DDL export</figcaption> </figure> Notes on SQL Developer:</p> Is the clearest</li> Many formats for data export (also CSV)</li> Extensively configurable</li> Export APEX App only with Commandline version SQLcl</li> </ul> Notes on PL/SQL Developer:</p> Little configurable</li> Disappointing for building a source code repository</li> </ul> Notes on Toad:</p> Two export options (at least). Either subdirectories per object type….</li> … or data</li> </ul> </li> Data only as insert statements</li> Extensively configurable, relatively unclear</li> </ul> None of the tools provides us with a ready-made, well-structured source code repository. You have to do more or less a lot of rework. That doesn’t mean it can’t be done, especially since sorting the objects into a reasonable structure should be a one-time job. Only in the case of exporting the APEX application one would like to have an automatic solution, because this export will have to be done more often in the future. This is in the nature of APEX as a declarative “low code framework”.</p> We develop the APEX application in the browser and the meta-data of the application resides in the APEX repository of the database. An export of the application can also be done with the browser, but in doing so you will only get a single large SQL file with the entire application. Desirable for a source code repository would be an export of the individual parts such as pages, shared components, plugins, etc.. This would then make it possible for the application to track how it has developed and also a search in the repository for e.g. a specific package call makes much more sense with a disassembled APEX application. In the past, you could use the APEX export splitter for this, basically a Java class that comes with every APEX version to this day. The disadvantage is that you have to take the directory structure as created by the splitter or you change it afterwards with local scripts on your PC. However, it would be nicer to be able to adjust the directory structure already during the export in order to be able to store all master scripts in a central folder of the repository.</p> Since APEX version 5.1.4 there is the APEX_EXPORT package for this purpose. With this package you can export either the whole script or the splitted version. The return format in both cases is a collection - each line of the collection contains a path specification for the respective export file and a CLOB field with the content. So you could edit this collection in an export script and adapt the path specifications to your own needs. This is exactly what the tool package PLEX, published as open source by the author of this article, does - the name stands for PL</strong></em>/SQL Ex</strong></em>port Utilities. Furthermore, PLEX can answer yes to any of the above questions about DDL export - no wonder, it was designed for that.</p> Quickstart: Package PLEX</h2> PLEX aims to be able to select the first version of the repository as a BLOB with a simple query. After saving the BLOB to the local directory system on the PC with the extension .zip you can unpack it and find in the generated directory structure script examples for future semi- or fully automatic exports and deployments via shell command or SQLPlus. A possible first export could look like shown in Listing 1.</p> with</span></span> function</span> backapp </span>return</span> blob </span>is</span></span> begin</span></span> return</span> plex.to_zip(plex.backapp(</span></span> p_app_id </span>=></span> 100</span>,</span></span> / If null, we simply skip the APEX app export.</span></span> Parameter only available if APEX installed. /</span></span> p_include_object_ddl </span>=></span> true,</span></span> / If true, include DDL of current user/schema and</span></span> all its objects. /</span></span> p_include_templates </span>=></span> true,</span></span> / If true, include templates for README.md, export</span></span> and install scripts. /</span></span> p_include_runtime_log </span>=></span> true,</span></span> / If true, generate file plex_backapp_log.md with</span></span> runtime statistics. /</span></span> p_include_data </span>=></span> true,</span></span> / If true, include CSV data of each table. /</span></span> p_data_max_rows </span>=></span> 1000</span>,</span></span> / Maximum number of rows per table. /</span></span> p_data_table_name_like </span>=></span> 'DEMO_PRODUCT_INFO,DEMO_STATES'</span></span> / A comma separated list of like expressions to filter</span></span> the tables. Example: 'EMP%,DEPT%' will be translated to</span></span> "where ... and (table_name like 'EMP%' escape '\' or</span></span> table_name like 'DEPT%' escape '\')". /</span></span> ));</span></span> end</span> backapp;</span></span> </span> select</span> backapp </span>from</span> dual;</span></span></code></pre> Listing 1: Possible first export</figcaption> </figure> Since PLEX has boolean parameters, we use an inline function in the with clause. If you use a database version smaller than 12c, you can create an auxiliary function similar to the example. PLEX has some more parameters to configure e.g. the APEX App Export. More details can be found on the official project page at github.com/ogobrecht/plex</a>. Depending on the size of the schema and the APEX app, this initial call can take anywhere from a few seconds to several minutes. This is because in the background for each object the package DBMS_METADATA is used to generate the DDL of the object. If you are interested in runtime information, you will find a log file in the main directory of the export with information about what PLEX did and how long the respective steps took. PLEX itself now exists in a first version - suggestions for improvements or bug reports are welcome and can be reported as issues on the project page.</p> From here on it strongly depends on the needs of the respective project if and how often you run a DDL export. The normal way should be a regular export of the APEX application. Everything else should be processed locally from now on and as a result there should be no need for an export. However, as mentioned at the beginning, there may be situations where you regularly export all schema objects in order to document the application. However, this should only be seen as an intermediate step towards a file-based way of working.</p> Generated code is a special case: So for example using Quick-SQL in APEX or generating table APIs. Here one can consider exporting the generated code with appropriately configured object filters. For APIs you should consider versioning the generator instead of the generated code - but this depends entirely on whether the generated code is still being manually modified or not. Any manually generated code should be versioned.</p> Geschwindigkeit: Immer Skripte</h2> As you can already see from the script examples provided by PLEX, the goal should be to execute any deployment by script. This also includes the consideration to export the object DDL only at the very first time - because what happens e.g. when exporting table DDL? If you have written another column into the table script via Alter Table statement and made sure that it is restartable (see Listing 2), then another DDL export overwrites the Alter statement and our table script simply contains the new column listed in the Create Table statement. However, this is never executed again due to the restartability, since our table already exists.</p> begin</span></span> for</span> i </span>in</span> (</span></span> select</span> 'DEMO_STATES'</span> as</span> object_name</span></span> from</span> dual</span></span> minus</span></span> select</span> object_name</span></span> from</span> user_objects )</span></span> loop</span></span> execute immediate</span> q</span>'[</span></span> ----------------------------------------</span></span> create table "demo_states" (</span></span> "st" varchar2(30),</span></span> "state_name" varchar2(30)</span></span> )</span></span> ----------------------------------------</span></span> ]'</span>;</span></span> end loop</span>;</span></span> end</span>;</span></span> /</span></span> </span> begin</span></span> for</span> i </span>in</span> (</span></span> select</span> 'STATE_DESCRIPTION'</span> AS</span> column_name</span></span> from</span> dual</span></span> minus</span></span> select</span> column_name</span></span> from</span> user_tab_columns</span></span> where</span> table_name </span>=</span> 'DEMO_STATES'</span></span> and</span> column_name </span>=</span> 'STATE_DESCRIPTION'</span> )</span></span> loop</span></span> execute immediate</span> q</span>'[</span></span> ----------------------------------------</span></span> alter table demo_states add (</span></span> state_description varchar2(255)</span></span> )</span></span> ----------------------------------------</span></span> ]'</span>;</span></span> end loop</span>;</span></span> end</span>;</span></span> /</span></span></code></pre> Listing 2: Restartable script</figcaption> </figure> Thus, we now have a source code file that is no longer suitable for deployment. A pity actually, because until just now we could have simply called all object scripts with our master script and only the changes (like our new column) would really have been executed - a simple deployment and a clear history in the repository. This is contrasted with the automatic creation of diff scripts, which are sometimes not restartable and risk data loss even with slightly more complicated schema changes. One cannot avoid checking the generated scripts for correctness and making manual adjustments for any data migrations, even with very expensive tools of this class. No one takes the decision of which way to go away from you. Now we are in the middle of the discussion on how to deploy to the target system. PLEX can’t help here either - except that the package per se tries to extract all necessary object DDLs in a way that they can be restarted. Here we are in the wide field of individual needs of a project. The script examples supplied by PLEX must be adapted to the needs of the respective project in any case. For example, the author almost always distinguishes between scripts for the backend and the frontend - both for export and deployment.</p> Consideration: Directory structure in the repository</h2> How should you structure your repository? Here is a suggestion that has proven to be beneficial for me personally over time. Every project may have different ideas. I use a simple listing here - the first words of each list item represent the directory name, and any explanations are in parentheses after it:</p> app_backend (our schema DDL) constraints (one folder per object type)</li> package_bodies</li> packages</li> ref_constraints</li> sequences</li> tables</li> …</li> </ul> </li> app_data (tracking of master data, optional)</li> app_frontend (our APEX app) pages</li> shared_components</li> …</li> </ul> </li> docs (the documentation)</li> reports</li> scripts (all master scripts combined) logs (export and installation logs) export_app_100_from_MYSCHEMA_at_DEV_20190315_132542</li> install_app_100_into_MYSCHEMA_at_TEST_20190318_083756</li> …</li> </ul> </li> misc (frequently used scripts like recompile schema)</li> 1_export_app_from_DEV.bat (shell script with parameters for the actual SQL export script)</li> 2_install_app_into_TEST.bat (Shell script with parameters for the SQL deployment script)</li> 3_install_app_into_PROD.bat (shell script with parameters for the custom SQL deployment script)</li> export_app_custom_code.sql (export master script with customizations)</li> install_app_custom_code.sql (Deployment Master Script with individual customizations per release)</li> install_backend_generated_by_plex.sql (called from master script)</li> install_frontend_generated_by_apex.sql (called by the master script)</li> </ul> </li> tests (unit and frontend tests)</li> README.md (general information about the project and links to the documentation)</li> </ul> The repository should be logically structured and not contain extremely deep directory structures to make it easier to use. All master scripts are united in one script folder, also the installation script generated by APEX for the frontend. All master scripts should store their respective feedback in corresponding log files in the logs subfolder during use - this makes exports of source code and deployments to target systems traceable.</p> Outlook: CI/CD & Schema Migration</h2> The outlined discussion of deployment already hints at it: Simply setting up a source code repository and you’re done is not the solution. That was just the first step to warm up. Once you have structured everything and made it executable via script, you can automate the deployment in the second step. The content of this discussion is far beyond the scope of this article, which focuses on the first step. Therefore, the author suggests everyone not to stop after this first step and to consider further steps. Martin Fowler, for example, has written a very good basic article on the subject - he considers every change to the database as a migration. For further study, the following entry points should therefore be mentioned:</p> Antti Kirmanen: Git vs. Subversion (SVN): Welches Versionskontrollsystem sollten Sie nutzen?</a></li> Martin Fowler: Evolutionary Database Design</a></li> Samuel Nitsche: There is no clean (database) development without Version Control</a></li> “One does not simply update a database” – migration based database development</a></li> </ul> </li> Blain Carter: Tips to help PL/SQL developers get started with CI/CD</a></li> CI/CD for Database Developers – Export Database Objects into Version Control</a></li> </ul> </li> Jeff Smith: 19.X SQLcl Teaser: LIQUIBASE</a></li> </ul> Happy version controlling :-)</p> Ottmar</p> New Major Version of PL/SQL Export Utilities Available 2019-06-26T00:00:00+00:00 PLEX is a PL/SQL package with export utilities - see this post</a> what it can do for you. In the past it was dependent on APEX for two reasons: The APEX_EXPORT package to allow exporting an APEX app and the APEX_ZIP package for compressing the resulting file collection to a zip file.</p> After the first public release of PLEX, I got a comment like “Nice tool, but I don’t use APEX”. I thought I should make PLEX independent from APEX to allow at least the export of all the schema DDL and also the table data. Some weeks ago Jürgen Schuster contacted me to try out PLEX for an switch from Subversion to Git for one of his customer projects. It turns out that PLEX was not able to export the schema DDL because of errors from DBMS_METADATA regarding many Java classes. And the second issue was that one of their machines had no APEX installed. This was the final trigger for me to rework the implementation of PLEX.</p> PLEX is now independent from an APEX installation. Thanks to Anton Scheffer for his as_zip package</a>. PLEX reuses the methods for compressing files and uses conditional compilation to include/exclude functionality depending on APEX. Because PLEX cannot rely on APEX anymore, the return type for the methods backapp</strong> and queries_to_csv</strong> has changed from apex_t_export_files</code> to plex.tab_export_files</code>. This was unavoidable and if you have existing scripts and you want to update to the new PLEX version, you have to change the variables based on that return type. A simple search and replace will normally do the trick. This breaking change in the PLEX API is also the reason for a new major version.</p> The other improvement is better error handling and logging. Per default you will find two files in an export collection: plex_runtime_log and plex_error_log. The former is for an overview what happened and how long it took and the latter is for detailed error messages. If you encounter errors and you think that should not happen and is a problem of PLEX, then please open an issue on GitHub</a>. I was able to export the schema APEX_19_0100 with around 5000 objects on my dev machine…</p> Project on Github</a></li> Download latest version</a></li> </ul> Happy exporting :-)</p> Ottmar</p> New Version of D3 Force Network Chart Available 2019-06-09T00:00:00+00:00 </p> First of all: Thanks to the community!</p> From time to time someone has new ideas or questions about the D3 network chart. Some time ago I thought my D3 implementation is more or less feature complete. But in the last weeks I worked again on the code to bring in new feature wishes from the community.</p> The most visible one in the new version 3.1 is the ability to add labels to links - see the links between Allen, Blake and Ward in the screenshot above. The labels are positioned in a way that they are always readable - this is the reason why they appear sometimes above and sometimes below a link line (they are rotated, if needed). The nice thing is that you can also read both labels when you have links in both directions between two nodes. The other benefit of having labels on links is that they extend the area for the link click and hover events. This improves the usabiltiy in cases where the links should show a tooltip when they are hovered or some action should be executed when they are clicked. This is normally not so easy because the links are so narrow.</p> There are also new helper methods and one new option - for the full change log see the project on GitHub</a> and for online demos either the standalone version in the docs</a> or the APEX plug-in version on apex.oracle.com</a>.</p> Happy networking :-)</p> Ottmar</p> PLOC - Convert PL/SQL Code into Markdown Docs 2019-01-10T00:00:00+00:00 </p> Some months ago I had to write a README file with Markdown syntax for a PL/SQL open source project. I changed function and procedure parameters of the package often during the implementation. Therefore, maintaining the documentation was no fun in the beginning. Maybe you know this from your own (payed) projects ;-)</p> Normally in the business world I see minimalistic comments in the code and a dedicated documentation independent of the code. As you can imagine, the latter is sometimes - let’s say “out of date”, if it exists at all …</p> I don’t want to discuss the reasons for this here - you all know several ones. And I also do not have a general solution for code documentation. But I have some ideas what is important for me:</p> The base of a good PL/SQL code documentation is a good code structure No single functions and procedures - packages with meaningful names are the way to go</li> </ul> </li> As I have the source code under version control in local files on my computer I need a command line tool for the doc generation - there is no need for a database connection The CLI should be able to process multiple files (packages)</li> For my small open source project I have only one package and I want to define a document name different then the package name, so I need a CLI parameter for that</li> </ul> </li> Comments should be written directly after the package/function/procedure signature (in the package spec, optionally it should also work for triggers and types) I want to use Markdown for the comments - this allows me to write also bigger text parts and code examples</li> </ul> </li> I don’t want to describe all parameters in a JavaDoc manner PL/SQL is strong typed, everything I need to know is visible in the signature of a function or procedure</li> Replicating the parameters for documentation is error prone and does not follow the DRY principles</li> With meaningful parameter names you need often no or only short comments - these can be placed directly after the parameters as single line comments</li> Long descriptions can be written in the Markdown main comment</li> </ul> </li> The package/function/procedure signature should be automatically included in the output as a code block for reference</li> Output should also be Markdown - this can be easily read by humans and further processed by machines Only documented functions/procedures should be included in the output to keep the focus on the important stuff</li> I want to have an automatic table of contents in each generated document</li> With the help of Pandoc</a> many formats like Word, RTF, PDF or HTML can be generated (for GitHub I only need Markdown)</li> </ul> </li> </ul> There are already several open source projects available for PL/SQL code documentation generation:</p> PLDoc</a></li> OraDoclet</a></li> SchemaSpy</a></li> Natural Docs</a></li> PL/SQL to Markdown Documenter</a></li> </ul> Only the last one is able to generate Markdown docs, but the input has to be in JavaDoc. All the others can produce HTML output and often some other formats like RTF, Word or PDF. Natural Docs has a Markdown like simple text syntax, but also no Markdown output. Some of the tools can automatically document a whole schema - that is not the focus of PLOC. Most comparable is the intention of PL/SQL to Markdown Documenter.</p> So I had to do it by myself. I called it PLOC (PLDoc was already used, so I skipped the D…) and it is implemented in JavaScript as a npm package and currently used by me for the following two projects: PL/SQL Export Utilities</a> and Oracle Table API Generator</a>. It is very lightweight - the main code without the CLI has currently less characters as the Markdown source of this blog post. Maybe it is also useful for you. If not, have a look at the other mentioned projects.</p> More infos about PLOC</a> itself can be found on the project page</a>.</p> Happy coding and documenting :-)</p> Ottmar</p> Table API Generator v0.5.0 Available 2018-12-26T00:00:00+00:00 Short before Christmas André and I released version 0.5.0 of our PL/SQL table API generator</a>. It took a long time to come to this point. One of the reasons behind was the fact that we had to rework some internals to support feature wishes from the community. Here the most important features of the new version:</p> Support for multicolumn primary keys</li> Support for 12c long identifier names</li> More control over the API features and name (10 new parameters)</li> Insert methods: Can have the dictionary column defaults</li> Can return the whole row instead of the primary key</li> </ul> </li> You can exclude columns (e.g. audit columns) for inserts and updates</li> Improved template engine: Generated code better readable because of formatted parameter lists</li> Tables with many columns no longer breaking character limits</li> </ul> </li> Special methods for testing and dummy data generation</li> Also see the full changelog on the project site</a></li> </ul> Happy new year and happy coding :-)</p> Ottmar</p> New Major Version of D3 Force Network Chart Available 2018-12-02T00:00:00+00:00 </p> Since APEX version 5 was published with the Universal Theme, I was thinking that I needed something for better responsiveness of the graph. Although it was possible to configure the graph to use the DOM parent width, the responsiveness was not so good because of situations where the window size does not change (we use the window resize event to trigger the change of the width) but the available space for the graph changes. How can the graph react to this?</p> The problem is that a network chart is not a simple image which can be resized according to the available space. The graph is an SVG with multiple layers to handle the zoom and pan, the lasso, a legend that needs to be in a fixed size… As you can see, there must be a procedural solution. And for this we need a trigger, which always fires when the (possible) size of the graphs parent DOM element changes.</p> Fortunately, there is a W3C draft for a resize observer</a> and a polyfill</a> available.</p> To put the things together, I had to make breaking changes on the API (the reason for a new major version):</p> I made the API zoom methods independent of the option zoomMode</code> and set the default to true for the following options:</p> zoomToFitOnForceEnd</code> (was false in the past)</li> zoomToFitOnResize</code> (new option)</li> keepAspectRatioOnResize</code> (new option)</li> </ul> When setting the option useDomParentWidth</code> to true together with the previous mentioned defaults, you can achieve a responsiveness like with images set to width 100%. Try it out by resizing your browser window.</p> For the full change log see the project on GitHub</a> and for a demo the app on apex.oracle.com</a> (which currently does not use the universal theme, new demo app is work in progress…).</p> Happy networking :-)</p> Ottmar</p> PLEX - PL/SQL Export Utilities 2018-08-26T00:00:00+00:00 PLEX is a standalone PL/SQL package with export utilities. It was created to be able to quickstart version control for existing (APEX) apps. It currently has two main functions called backapp</code> and queries_to_csv</code>. queries_to_csv</code> is used by BackApp as a helper function, but its functionality is also useful as a standalone. This post is all about BackApp, which has the following features:</p> Export the app definition of an APEX app (splitted files and optional single SQL file)</li> Export all ORDS modules from the current schema</li> Export all object DDL from the current schema</li> Export table data into CSV files</li> Provide basic script templates for export/import of whole app for DEV, TEST and PROD</li> Everything in a (hopefully) nice directory structure ready to use with version control</li> Return value is a file collection of type plex.tab_export_files (it was apex_t_export_files before PLEX version 2) for further processing Each file in the collection is represented by a record with two columns name</code> of type VARCHAR2(255), which is in fact the file path</li> contents</code> of type CLOB</li> </ul> </li> You can optionally zip the file collection with the helper function to_zip</code></li> Also see the my previous post on how to handle the apex_t_export_files type returned by the APEX_EXPORT package with SQLPlus</a></li> </ul> </li> </ul> Getting Started</h2> Download the latest code</a></li> Run the provided install script plex_install.sql</code> (provides compiler flags) in your desired schema - could also be a central tools schema, don’t forget grant execute on plex to xxx</code></li> Startup your favorite SQL Tool, connect to your app schema and fire up the following query:</li> </ol> select</span> plex.to_zip(plex.backapp(p_app_id </span>=></span> yourAppId)) </span>from</span> dual;</span></span></code></pre> Save the resulting BLOB file under a name with the extension .zip</code> and extract it to a local directory of your choice. You will find this directory structure and files:</p> - app_backend (only, when p_include_object_ddl is set to true, see next example)</span></span> - package_bodies</span></span> - packages</span></span> - tables</span></span> - ref_constraints</span></span> - ...</span></span> - app_data (only, when p_include_data is set to true)</span></span> - app_frontend (for the apex app files without subfolder fxxx)</span></span> - pages</span></span> - shared_components</span></span> - ...</span></span> - app_web_services (only, when p_include_ords_modules is set to true)</span></span> - docs</span></span> - scripts</span></span> - logs</span></span> - templates</span></span> - 1_export_app_from_DEV.bat</span></span> - 2_install_app_into_TEST.bat</span></span> - 3_install_app_into_PROD.bat</span></span> - export_app_custom_code.sql</span></span> - install_app_custom_code.sql</span></span> - install_backend_generated_by_plex.sql</span></span> - install_frontend_generated_by_apex.sql</span></span> - install_web_services_generated_by_ords.sql</span></span> - tests</span></span> - plex_README.md</span></span> - plex_runtime_log.md</span></span></code></pre> If you like, you can fully configure your first export into the zip file. The plex.backapp</code> method has boolean parameters, so you need to use an inline function in a pure SQL context. You can also use an anonymous PL/SQL block or you create a small SQL wrapper for the method like the inline function of the example. All parameters are optional and listed here with their default values:</p> -- Inline function because of boolean parameters (needs Oracle 12c or higher).</span></span> -- Alternative create a helper function and call that in a SQL context.</span></span> WITH</span></span> FUNCTION</span> backapp </span>RETURN</span> BLOB </span>IS</span></span> BEGIN</span></span> RETURN</span> plex.to_zip(plex.backapp(</span></span> -- All parameters are optional and shown with their defaults</span></span> -- APEX App (only available, when APEX is installed):</span></span> p_app_id </span>=></span> null</span>, </span>-- If null, we simply skip the APEX app export.</span></span> p_app_date </span>=></span> true, </span>-- If true, include export date and time in the result.</span></span> p_app_public_reports </span>=></span> true, </span>-- If true, include public reports that a user saved.</span></span> p_app_private_reports </span>=></span> false, </span>-- If true, include private reports that a user saved.</span></span> p_app_notifications </span>=></span> false, </span>-- If true, include report notifications.</span></span> p_app_translations </span>=></span> true, </span>-- If true, include application translation mappings and all text from the translation repository.</span></span> p_app_pkg_app_mapping </span>=></span> false, </span>-- If true, export installed packaged applications with references to the packaged application definition. If FALSE, export them as normal applications.</span></span> p_app_original_ids </span>=></span> false, </span>-- If true, export with the IDs as they were when the application was imported.</span></span> p_app_subscriptions </span>=></span> true, </span>-- If true, components contain subscription references.</span></span> p_app_comments </span>=></span> true, </span>-- If true, include developer comments.</span></span> p_app_supporting_objects </span>=></span> null</span>, </span>-- If 'Y', export supporting objects. If 'I', automatically install on import. If 'N', do not export supporting objects. If null, the application's include in export deployment value is used.</span></span> p_app_include_single_file </span>=></span> false, </span>-- If true, the single sql install file is also included beside the splitted files.</span></span> p_app_build_status_run_only </span>=></span> false, </span>-- If true, the build status of the app will be overwritten to RUN_ONLY.</span></span> -- ORDS Modules (only available, when ORDS is installed):</span></span> p_include_ords_modules </span>=></span> false, </span>-- If true, include ORDS modules of current user/schema.</span></span> -- Schema Objects:</span></span> p_include_object_ddl </span>=></span> false, </span>-- If true, include DDL of current user/schema and all its objects.</span></span> p_object_type_like </span>=></span> null</span>, </span>-- A comma separated list of like expressions to filter the objects - example: '%BODY,JAVA%' will be translated to: ... from user_objects where ... and (object_type like '%BODY' escape '\' or object_type like 'JAVA%' escape '\').</span></span> p_object_type_not_like </span>=></span> null</span>, </span>-- A comma separated list of not like expressions to filter the objects - example: '%BODY,JAVA%' will be translated to: ... from user_objects where ... and (object_type not like '%BODY' escape '\' and object_type not like 'JAVA%' escape '\').</span></span> p_object_name_like </span>=></span> null</span>, </span>-- A comma separated list of like expressions to filter the objects - example: 'EMP%,DEPT%' will be translated to: ... from user_objects where ... and (object_name like 'EMP%' escape '\' or object_name like 'DEPT%' escape '\').</span></span> p_object_name_not_like </span>=></span> null</span>, </span>-- A comma separated list of not like expressions to filter the objects - example: 'EMP%,DEPT%' will be translated to: ... from user_objects where ... and (object_name not like 'EMP%' escape '\' and object_name not like 'DEPT%' escape '\').</span></span> p_object_view_remove_col_list </span>=></span> true, </span>-- If true, the outer column list, added by Oracle on views during compilation, is removed</span></span> -- Table Data:</span></span> p_include_data </span>=></span> false, </span>-- If true, include CSV data of each table.</span></span> p_data_as_of_minutes_ago </span>=></span> 0</span>, </span>-- Read consistent data with the resulting timestamp(SCN).</span></span> p_data_max_rows </span>=></span> 1000</span>, </span>-- Maximum number of rows per table.</span></span> p_data_table_name_like </span>=></span> null</span>, </span>-- A comma separated list of like expressions to filter the tables - example: 'EMP%,DEPT%' will be translated to: where ... and (table_name like 'EMP%' escape '\' or table_name like 'DEPT%' escape '\').</span></span> p_data_table_name_not_like </span>=></span> null</span>, </span>-- A comma separated list of not like expressions to filter the tables - example: 'EMP%,DEPT%' will be translated to: where ... and (table_name not like 'EMP%' escape '\' and table_name not like 'DEPT%' escape '\').</span></span> -- General Options:</span></span> p_include_templates </span>=></span> true, </span>-- If true, include templates for README.md, export and install scripts.</span></span> p_include_runtime_log </span>=></span> true, </span>-- If true, generate file plex_runtime_log.md with detailed runtime infos.</span></span> p_include_error_log </span>=></span> true, </span>-- If true, generate file plex_error_log.md with detailed error messages.</span></span> p_base_path_backend </span>=></span> 'app_backend'</span>, </span>-- The base path in the project root for the Schema objects.</span></span> p_base_path_frontend </span>=></span> 'app_frontend'</span>, </span>-- The base path in the project root for the APEX app.</span></span> p_base_path_web_services </span>=></span> 'app_web_services'</span>, </span>-- The base path in the project root for the ORDS modules.</span></span> p_base_path_data </span>=></span> 'app_data'</span>)); </span>-- The base path in the project root for the table data.</span></span> END</span> backapp;</span></span> SELECT</span> backapp </span>FROM</span> dual;</span></span></code></pre> ATTENTION: Exporting all database objects can take some time. I have seen huge runtime differences from 6 seconds for a small app up to several hundred seconds for big apps and/or slow databases. This is normally not the problem of PLEX. If you are interested in runtime statistics of PLEX, you can inspect the delivered plex_runtime_log.md</code> in the root directory. Also, the possibility to export the data of your tables into CSV files does not mean that you should do this without thinking about it. The main reason for me to implement this feature was to track changes on catalog tables by regularly calling this export feature with a sensitive table filter and max rows parameter as catalog data is often relevant in business logic.</p> If you have organized your app into multiple schemas as described in The Pink Database Paradigm</a>, you may need to export database objects from more then one schema. This is no problem for plex.backapp</code> as all parameters are optional - you can simply logon to your second or third schema and extract only the DDL for these schemas by omitting the p_app_id</code> parameter and setting p_include_object_ddl</code> to true</code>. Then unload the DDL files into a different directory - for example app_backend_schemaName</code>.</p> A last word: you should inspect all the exported files and scripts and check if this solution can work for you. If not, please let me know what is missing or what should be done in a different way …</p> Feedback is welcome - simply create a new issue</a> at the GitHub project page</a></p> Next Steps</h2> It is up to you how you organize the version control repository and how often you export your APEX app or object DDL. I would follow the files first approach and extract the object DDL only ones to have a starting point. The APEX application needs regular exports - if you like, you can automate this.</p> Following the files first approach is sometimes not easy when you are using low code tools like Quick SQL</a> and Blueprint</a> in APEX or code generators like OraMUC’s Table API Generator</a>. There could be a need to regularly extract (maybe unknown) objects (not created by yourself) into version control to understand and document what you got from others (people or generators)…</p> If the directory structure provided by PLEX does not match your needs - no problem - you can align it. Simply loop over the returned file collection and do your necessary work - here an example:</p> DECLARE</span></span> l_files plex.tab_export_files; </span>-- before PLEX v2 it was apex_t_export_files</span></span> BEGIN</span></span> l_files :</span>=</span> plex.backapp(p_app_id </span>=></span> 100</span>);</span></span> FOR</span> i </span>IN</span> 1</span>..l_files.count</span> LOOP</span></span> -- relocate APEX app files from app_frontend to app_ui</span></span> IF</span> l_files(i).</span>name LIKE</span> 'app_frontend/%'</span> THEN</span></span> l_files(i).</span>name</span> :</span>=</span> replace</span>(l_files(i).</span>name</span>, </span>'app_frontend/'</span>, </span>'app_ui/'</span>);</span></span> l_files(i).contents :</span>=</span> replace</span>(l_files(i).contents, </span>'prompt --app_frontend/'</span>, </span>'prompt --app_ui/'</span>);</span></span> END IF</span>;</span></span> -- correct file links in install script</span></span> IF</span> l_files(i).</span>name</span> =</span> 'scripts/install_frontend_generated_by_apex.sql'</span> THEN</span></span> l_files(i).contents :</span>=</span> replace</span>(l_files(i).contents, </span>'@../app_frontend/'</span>, </span>'@../app_ui/'</span>);</span></span> END IF</span>;</span></span> END LOOP</span>;</span></span> </span> -- more alignments...</span></span> END</span>;</span></span></code></pre> For unloading the resulting file collection with SQLPlus, please have a look in the scripts/templates</code> folder of your export - there are examples to do this. See also my previous post on how to handle the apex_t_export_files type returned by the APEX_EXPORT package with SQLPlus</a>.</p> Some people prefer to devide their DDL scripts into the two categories restartable</strong> (like packages) and run once</strong> (like tables). Others like to have their scripts in a way that they are always restartable and the DDL script itself takes care about doing the work only once when needed. The advantage of the second way is that your backend install/deployment script is always the same and it simply calls all objects DDL scripts.</p> There is no right or wrong in doing it this or that way - each project/team has its specific requirements and history. The important thing is, that you start to use a version control system to be able to log your changes and document your code.</p> By the way - PLEX provides script templates and object DDL that follows the second approach: You can always have the same install/deployment script and the DDL scripts are restartable - check it out by looking in one of your exported table DDL scripts.</p> You are now at the point where PLEX can’t do anything more for you. If you like to export your object DDL scripts more often, you have to find a way to be able to protect some of your scripts against overwriting. Imagine you had to add two columns to a table and you provided a restartable alter statement for this in the existing DDL script. If you export this table script the next time with PLEX (or with dbms_metadata.get_ddl, which is used in the background), your alter statements are gone and the new columns are simply listed in the create table statement. With this script you are not be able to deploy your changes to TEST or PROD. One solution is to copy the original table script and name it e.g. EMPLOYEES.dev.sql</code>. In this script you maintain the restartable alter statements. If you run plex.backapp</code> again you are overwrite save. The script EMPLOYEES.sql</code> reflects your current table definition and can still be executed - it does nothing because the table is already existing. The script EMPLOYEES.dev.sql</code> reflects your development history and need to be added to your custom install/deployment script.</p> As you can see, PLEX can do only the basics for you. It is up to the developers how they manage their version control repository and how they do their deployments - there are thousends of ways to do it …</p> Inspirations / Further Reading</h2> Thanks are going to:</p> André Borngräber for his ability to think and discuss database topics in deep details</li> Blain Carter for his thoughts on CI/CD for Database Developers – Export Database Objects into Version Control</a></li> Markus Dötsch for the first BackApp tests and cross-reading the first version of this post</li> Martin D’Souza for his time and the interesting discussion about code generation and version control at the APEX Connect 2018 in Düsseldorf, Germany and his blog post Exporting APEX Application in SQLcl with Build Status Override</a> - plex.backapp</code> has now a parameter for this ;-)</li> Philipp Salvisberg for his thoughts on The Pink Database Paradigm</a></li> Samuel Nitsche for his thoughts on There is no clean (database) development without Version Control</a></li> Tim Hall for his article about Generating CSV Files</a></li> </ul> Thats It</h2> Hopefully PLEX is usefull for someone else.</p> Happy coding, apexing, version controlling :-)</p> Ottmar</p> APEX_EXPORT and Version Control 2018-07-25T00:00:00+00:00 Since years it has been possible to export an APEX app definition with the help of APEXExport, a Java utility delivered within the APEX install zip file. There is also the possibility to split the file into its components like pages, plugins and so on. There are some blog postings available how to do this - simply ask Google</a>. Also the Java based SQLcl has the capability to do the export of an APEX app directly.</p> So why bother with a different way to export and split an APEX application?</p> Since APEX 5.1.4 there is a new PL/SQL package APEX_EXPORT, which can be used to get a file collection of the application - one big file or the splitted ones. Unfortunately in the API docs</a> there is (as of this writing) only one example available how to export the single file within SQLPlus - no example to handle the splitted files.</p> But again, why discuss this if there are already options to do it?</p> Because of the possibility to modifiy the APEX_EXPORT file collection before fetching it into the file system. Imagine you have a different repository structure and the delivered file structure of the splitted files does not match your needs or you want to have all install files in one scripts directory of your repo and therefore need a relocation of the app install script. Another use case is to enrich the file collection with additional data or objects. This was possible in the past also with some postprocessing outside the database, but now we are able to do this within our DB session and PL/SQL. I have already started an open source project to leaverage these possibilities - more about this in my next post…</p> So how to do it?</p> First, I have to describe what the sctructure of each file in the collection is. Dead simple: a record type with two columns: name</code> of type VARCHAR2(255) which is in fact the file path and contents</code> of type CLOB.</p> The desired file structure:</p> - app_backend</span></span> - app_frontend (for the splitted files without subfolder fxxx)</span></span> - pages</span></span> - shared_components</span></span> - ...</span></span> - docs</span></span> - scripts</span></span> - logs</span></span> - temp_export_files.sql (our intermediate script file)</span></span> - export_frontend_from_DEV_20180722_2045.log (one of our export logs)</span></span> - ...</span></span> - export_frontend.bat (our OS shell script to start the export)</span></span> - export_frontend.sql (our export script)</span></span> - install_frontend.sql (the generated install file from apex_export)</span></span> - tests</span></span> - README.md</span></span></code></pre> Here comes the idea:</p> We create a script file to get the file collection, iterate over the collection and modify the content regarding our needs</li> To unload the files with the spool command in SQLPlus we need it accessible via SQL - therefore we put the files into a global temporary table</li> We need to create an intermediate script file to unload the files (select the clob content)</li> We also need to create host commands for the needed directories because the spool command does NOT create missing directories</li> We spool our progress to a log file for later reference</li> </ul> -- file: export_frontend.sql</span></span> </span> set</span> verify </span>off</span> feedback </span>off</span> heading </span>off</span></span> set</span> trimout </span>on</span> trimspool </span>on</span> pagesize </span>0</span> linesize </span>5000</span> long </span>100000000</span> longchunksize </span>32767</span></span> whenever sqlerror exit sql.sqlcode</span> rollback</span></span> </span> </span> -- https://blogs.oracle.com/opal/sqlplus-101-substitution-variables</span></span> define logfile </span>=</span> "logs/export_frontend_from_&2._&3._&4..log"</span></span> spool </span>"&logfile."</span> replace</span></span> </span> </span> prompt</span></span> prompt </span>Start</span> frontend export </span>for</span> app &</span>1</span>. </span>on</span> &</span>2</span>.</span></span> prompt </span>==================================================</span></span> prompt </span>Create global</span> temporary </span>table</span> temp_export_files </span>if not</span> exist</span></span> BEGIN</span></span> FOR</span> i </span>IN</span> (</span>SELECT</span> 'TEMP_EXPORT_FILES'</span> AS</span> object_name </span>FROM</span> dual</span></span> MINUS</span></span> SELECT</span> object_name </span>FROM</span> user_objects) </span>LOOP</span></span> EXECUTE IMMEDIATE</span> q</span>'[</span></span> --------------------------------------------------------------------------------</span></span> CREATE GLOBAL TEMPORARY TABLE temp_export_files (</span></span> name VARCHAR2(255),</span></span> contents CLOB</span></span> ) ON COMMIT DELETE ROWS</span></span> --------------------------------------------------------------------------------</span></span> ]'</span>;</span></span> END LOOP</span>;</span></span> END</span>;</span></span> /</span></span> </span> </span> prompt Do the frontend export, relocate files </span>and save to</span> temporary </span>table</span></span> DECLARE</span></span> l_app_id pls_integer :</span>=</span> &</span>1</span>.;</span></span> l_files apex_t_export_files;</span></span> BEGIN</span></span> l_files :</span>=</span> apex_export.get_application(</span></span> p_application_id </span>=></span> l_app_id,</span></span> p_split </span>=></span> true,</span></span> p_with_date </span>=></span> true,</span></span> p_with_ir_public_reports </span>=></span> true,</span></span> p_with_ir_private_reports </span>=></span> false,</span></span> p_with_ir_notifications </span>=></span> false,</span></span> p_with_translations </span>=></span> true,</span></span> p_with_pkg_app_mapping </span>=></span> false,</span></span> p_with_original_ids </span>=></span> true,</span></span> p_with_no_subscriptions </span>=></span> false,</span></span> p_with_comments </span>=></span> true,</span></span> p_with_supporting_objects </span>=></span> 'Y'</span>);</span></span> FOR</span> i </span>IN</span> 1</span>..l_files.count</span> LOOP</span></span> -- relocate files to own project structure</span></span> l_files(i).</span>name</span> :</span>=</span> replace</span>(</span></span> l_files(i).</span>name</span>,</span></span> 'f'</span>\|\|</span>l_app_id</span>\|\|</span>'/application/'</span>,</span></span> '../app_frontend/'</span></span> );</span></span> -- correct prompts for relocation</span></span> l_files(i).contents :</span>=</span> replace</span>(</span></span> l_files(i).contents,</span></span> 'prompt --application/'</span>,</span></span> 'prompt --app_frontend/'</span></span> );</span></span> -- special handling for install file</span></span> IF</span> l_files(i).</span>name</span> =</span> 'f'</span>\|\|</span>l_app_id</span>\|\|</span>'/install.sql'</span> THEN</span></span> l_files(i).</span>name</span> :</span>=</span> 'install_frontend.sql'</span>;</span></span> l_files(i).contents :</span>=</span> replace</span>(</span></span> replace</span>(</span></span> l_files(i).contents,</span></span> '@application/'</span>,</span></span> '@../app_frontend/'</span></span> ),</span></span> 'prompt --install'</span>,</span></span> 'prompt --install_frontend'</span></span> );</span></span> END IF</span>;</span></span> END LOOP</span>;</span></span> FORALL i </span>IN</span> 1</span>..l_files.count</span></span> INSERT INTO</span> temp_export_files </span>VALUES</span> (</span></span> l_files(i).</span>name</span>,</span></span> l_files(i).contents</span></span> );</span></span> END</span>;</span></span> /</span></span> </span> </span> prompt </span>Create intermediate</span> script </span>file to unload</span> the </span>table</span> content </span>into</span> files</span></span> spool </span>off</span></span> set</span> termout </span>off</span> serveroutput </span>on</span></span> spool </span>"logs/temp_export_files.sql"</span></span> BEGIN</span></span> -- create host commands for the needed directories (spool does NOT create missing directories)</span></span> FOR</span> i </span>IN</span> (</span></span> WITH</span> t1 </span>AS</span> (</span></span> SELECT</span> regexp_substr(</span>name</span>, </span>'^((\w\|\.)+\/)+'</span>) </span>AS</span> dir</span></span> FROM</span> temp_export_files</span></span> )</span></span> SELECT DISTINCT</span></span> dir,</span></span> -- This is for Windows to create a directory and suppress warning if it exist.</span></span> -- Align the command to your operating system:</span></span> 'host mkdir "'</span> \|\|</span> replace</span>(dir,</span>'/'</span>,</span>'\'</span>) </span>\|\|</span> '" 2>NUL'</span> AS</span> mkdir</span></span> FROM</span> t1</span></span> WHERE</span> dir </span>IS NOT NULL</span></span> ) </span>LOOP</span></span> dbms_output.put_line(</span>'set termout on'</span>);</span></span> dbms_output.put_line(</span>'spool "&logfile." append'</span>);</span></span> dbms_output.put_line(</span>'prompt --create directory if not exist: '</span> \|\|</span> i.dir);</span></span> dbms_output.put_line(</span>'spool off'</span>);</span></span> dbms_output.put_line(</span>'set termout off'</span>);</span></span> dbms_output.put_line(i.mkdir);</span></span> dbms_output.put_line(</span>'-----'</span>);</span></span> END LOOP</span>;</span></span> </span> -- create the spool calls for unload the files</span></span> FOR</span> i </span>IN</span> (</span>SELECT</span> </span> FROM</span> temp_export_files) </span>LOOP</span></span> dbms_output.put_line(</span>'set termout on'</span>);</span></span> dbms_output.put_line(</span>'spool "&logfile." append'</span>);</span></span> dbms_output.put_line(</span>'prompt --'</span> \|\|</span> i.name);</span></span> dbms_output.put_line(</span>'spool off'</span>);</span></span> dbms_output.put_line(</span>'set termout off'</span>);</span></span> dbms_output.put_line(</span>'spool "'</span> \|\|</span> i.name</span> \|\|</span> '"'</span>);</span></span> dbms_output.put_line(</span>'select contents from temp_export_files where name = '''</span> \|\|</span> i.name</span> \|\|</span> ''';'</span>);</span></span> dbms_output.put_line(</span>'spool off'</span>);</span></span> dbms_output.put_line(</span>'-----'</span>);</span></span> END LOOP</span>;</span></span> END</span>;</span></span> /</span></span> spool </span>off</span></span> set</span> termout </span>on</span> serveroutput </span>off</span></span> spool </span>"&logfile."</span> append</span></span> </span> </span> prompt </span>Call</span> the </span>intermediate</span> script </span>file to save</span> the files</span></span> spool </span>off</span></span> @logs</span>/</span>temp_export_files.sql</span></span> set</span> termout </span>on</span> serveroutput </span>off</span></span> spool </span>"&logfile."</span> append</span></span> </span> </span> prompt </span>Delete</span> files </span>from</span> the </span>global</span> temporary </span>table</span></span> COMMIT</span>;</span></span> </span> </span> prompt </span>==================================================</span></span> prompt Export DONE :</span>-</span>)</span></span> prompt</span></span></code></pre> To run this script file on your operating system, you need a shell script to call it. Here is an example for Windows:</p> rem</span> file: export_frontend.bat</span></span> </span> echo off</span></span> setlocal</span></span> set</span> systemrole</span>=</span>DEV</span></span> set</span> connection</span>=</span>localhost:1521/orcl</span></span> set</span> schema</span>=</span>HR</span></span> set</span> app_id</span>=</span>100</span></span> set</span> areyousure</span>=</span>N</span></span> </span> rem</span> align delimiters to your os locale</span></span> for</span> /f </span>"tokens=1-3 delims=. "</span> %%</span>a</span> in</span> (</span>'date /t'</span>)</span> do</span> (</span>set mydate=%%</span>c</span>%%</span>b</span>%%</span>a</span>)</span></span> for</span> /f </span>"tokens=1-2 delims=:"</span> %%</span>a</span> in</span> (</span>'time /t'</span>)</span> do</span> (</span>set mytime=%%</span>a</span>%%</span>b</span>)</span></span> </span> :</span>PROMPT</span></span> echo</span>.</span></span> echo</span>.</span></span> set</span> /p areyousure</span>=</span>Export %schema% app %app_id% from %systemrole% (Y/N)?</span></span> </span> if</span> /i %areyousure%</span> neq</span> y </span>goto</span> END</span></span> set</span> NLS_LANG</span>=</span>AMERICAN_AMERICA.AL32UTF8</span></span> set</span> /p password</span>=</span>Please enter password for %schema% on %systemrole%:</span></span> echo</span> exit </span>\|</span> sqlplus -S %schema%/%password%</span>@</span>%connection%</span> ^</span></span> @</span>export_frontend.sql </span>^</span></span> %app_id%</span> ^</span></span> %systemrole%</span> ^</span></span> %mydate%</span> ^</span></span> %mytime%</span></span> </span> :</span>END</span></span> pause</span></span></code></pre> There are better ways then to ask for a password at runtime - but that is not the focus of this post. Also, you may not want to keep the command line open until the user presses a key in a fully automated setup. This is only to be able to see any errors before the shell window closes when opened via double click on the file.</p> Hope this helps someone.</p> Happy exporting :-)</p> Ottmar</p> Show Today Button in APEX Datepicker 2018-07-22T00:00:00+00:00 Maxime Tremblay has already described here</a> how to customize the datepicker.</p> He had to deal with missing classes after changing options. I found this week a different way to customize the datepicker without loosing classes added by APEX: Simply by setting datepicker defaults on page load before any datepicker is initialized.</p> In my case I wanted to enable the button panel which shows two buttons - one to go to the current month which is called Today</code> and a second one to close the datepicker which is called Done</code> I only wanted the today button, so I had to provide beside the JavaScript or dynamic action some CSS to hide the second one:</p> apex</span>.</span>jQuery</span>(document)</span>.</span>ready</span>(</span>function</span>(){</span></span> $</span>.</span>datepicker</span>.</span>setDefaults</span>(</span>{</span> 'showButtonPanel'</span>:</span> true</span> }</span>)</span>;</span></span> }</span>)</span>;</span></span></code></pre>body</span> .</span>ui-datepicker-close</span> {</span> display</span>:</span> none</span> !important</span>; }</span></span></code></pre> </p> Just for reference: The jQuery UI datepicker widget can be tested here</a>. Be careful to pick the correct version of the jQuery UI API docs for your specific APEX version.</p> Happy coding :-)</p> Ottmar</p> HydeBar 2017-11-29T00:00:00+00:00 Earlier this year I wrote at the end of this post</a>:</p> PS: In the meantime I switched complete to Markdown and Jekyll</a>, a static site generator - but this is another story for another post …</p> </blockquote> As always - it took a bit longer then expected. In the meantime I was fiddling around with Jekyll and modifying the default theme Minima to my needs. Then I stumbled over the JSDoc theme Minami</a> and the Liquid docs site</a> - both with nice sidebars. I wanted to have such a nice sidebar for my dev blog and began to look deeper how it was implemented. To my surprise the sidebar from the JSDoc theme Minami was complete CSS driven - cool.</p> It took me a while to full understand how it was working. With this knowledge and the design inspirations I was able to improve the Jekyll default theme to a sidebar template. As a bonus I integrated the HTML based slideshow library Reveal.js - you can write now your blog posts and slides with markdown.</p> The nice thing about blogging with markdown is, that you don’t need to switch your tools - you use the usual ones from the daily programming job. It goes even better when you use GitHub: You commit your changes and GitHub takes care about generating your blog content from your sources. And the result is a fast and secure static site.</p> I had a accepted talk at the DOAG Conference + Exhibition 2017</a> in the stream “Strategy & Business Practices” about blogging for developers with Jekyll. Unfortunately short before I had an accident and broke my right upper arm :-(</p> My colleague Markus Dötsch was taking over the presentation and I concentrated on delivering the free time project “Jekyll sidebar template”. This was the working title before I renamed it to “HydeBar” - a play with “Dr. Jekyll and Mr. Hyde” and the fact, that it is a sidebar template.</p> Yesterday I released version 1.0.0. You can start blogging for free in five minutes with this quick start guide</a>. The online demo</a> is serving as template, documentation and showcase. The mentioned slides from the DOAG Conference</a> are included in this demo (in German, sorry…).</p> Happy blogging :-)</p> Ottmar</p> Form Behaviour - Bug or Feature? 2017-09-26T00:00:00+00:00 Today I stumbled over this strange behaviour of an APEX page: The page was always submitted when I pressed the enter key in a search field to force an AJAX refresh on a report.</p> The item property to submit the page on pressing enter was switched off. I asked my collegue Markus, who had this error also some time ago (I could not remember the solution, although he told me about this issue). The reason was that the search field was the only input item on that page.</p> After a short Google search I found the “root cause” for this “bug”, or should I say “feature”? This behaviour is described in the HTML 2.0 specification (section 8.2)</a>.</p> To workaround this you can simply create a second input item and hide it by setting the “HTML Form Element Attributes” to style="display:none;"</code>. This was also posted by Denes Kubicek</a> back in 2008.</p> I like the idea of one of the Stack Overflow comments</a> to put anywhere into the form (in APEX in fact the whole page, e.g. the header or footer of a region) this HTML snippet and not to create an “useless” page item:</p> <</span>input</span> type</span>=</span>"text"</span> name</span>=</span>"StackOverflow1370021"</span> style</span>=</span>"</span>display:none;"</span> /></span></span></code></pre> It is worth to mention that I run the newest Chrome browser and this “feature” is still available…</p> Happy coding :-)</p> Ottmar</p> Oracle DB 11gXE Install File Swap Check Disabler 2017-06-13T00:00:00+00:00 Many people have problems to install Oracle 11XE in a Docker environment because the install file checks the available swap space in the container. In a container environment this fails often - see here</a> or here</a>, because the swap space is optimized for the entire stack and not controlled from within the operating system of the container.</p> We have to disable the swap space check in the installation file. I wrote another blog post about this</a>. The problem is here, that you need a Linux based system to do the necessary steps. Under Windows you have no chance and you have to do it by yourself because for license reasons everyone has to download his/her own copy of the install file from Oracle OTN</a>.</p> I came up with the idea to simply do all the steps in a Docker container under the same Linux (oraclelinux:7-slim) which is later on needed with the official Oracle Docker file</a> for an XE instance. With this solution you are able to prepare the install file more or less automatically under every operating system, which can run Docker - also under Windows. For more Details see the project on GitHub</a>.</p> Happy installing :-)</p> Ottmar</p> jQuery Plugin svg2img 2017-04-03T00:00:00+00:00 SVG based charts and visualizations are often used these days. But what if you want to use your browser inline SVGs, generated with some sort of JavaScript and fancy styled with CSS, offline - maybe in a presentation, send by email or printed out large scaled?</p> You can create a screenshot, of course. But does this look nice when it comes to scaling? You can try to copy the HTML code of the SVG and wrap it into a correct SVG container, but then you will loose in the most cases the stylings, as normally not all CSS definitions are directly attached to the SVG elements. You can also use a browser extension like this one for Google Chrome</a>, but then you depend on the browser and the extension.</p> There is now a declarative way with the help of a jQuery plugin, which is 100% client based:</p> jQuery</span>(</span>'#example-graph'</span>)</span>.</span>svg2img</span>()</span>;</span></span></code></pre> There is one default option - you can redeclare it:</p> jQuery</span>.</span>fn</span>.</span>svg2img</span>.</span>defaults</span> =</span> {</span></span> debug</span>:</span> false</span> // write debug information to console</span></span> };</span></span></code></pre> You can also set the debug option at runtime:</p> $</span>(</span>'#example-graph'</span>)</span>.</span>svg2img</span>(</span>{</span>debug</span>:</span>true</span>}</span>)</span>;</span></span></code></pre> The plugin checks if your selector is a SVG element. If this is not the case, it searches under each selector element for all SVGs and exports it. The name(s) of the file(s) are automatically derived from the element ID (or parent element ID) or set to export</code>. The current date and time is appended to the file name, for example example-graph-20170403-154653.svg</code>.</p> You can download all SVGs from one page by setting the selector to the document, body or svg - as you like it. There is no need for a each()</code> call, svg2img</code> works on the whole selection and is chainable:</p> $</span>(</span>"svg"</span>)</span>.</span>svg2img</span>()</span>.</span>css</span>(</span>"border"</span>,</span> "1px solid red"</span>)</span>;</span></span></code></pre> If you want to use anchors to provide export links then it is recommended to prevent the default action like so:</p> <</span>a</span> href</span>=</span>""</span> onclick</span>=</span>"</span>event</span>.</span>preventDefault</span>()</span>;</span> $</span>(</span>'#example-graph'</span>)</span>.</span>svg2img</span>()</span>;</span>"</span>></span>SVG</span></</span>a</span>></span></span></code></pre> Otherwise Firefox and IE failing to save. One last thing: Safari has currently (as of this writing) problems with the underlaying savAs()</code> implementation and tries to open the images in a new tab with or without success. See also this issue on GitHub</a>.</p> UPDATE 2017-04-14: After update to macOS Sierra 10.12.4 Safari 10.1 works also - but is as slow as before - collecting CSS styles runs on my MacBook 2800ms in Safari, 20ms in Google Chrome :-( I stopped to support other image formats then SVG since the canvas export behind the scenes was not really working in too many browsers and the core feature was and is to convert inline SVGs to standalone SVG images.</p> The project is hosted on GitHub</a> and MIT licensed.</p> Happy SVG exporting :-)</p> Ottmar</p> Pitfalls with Oracle 11g XE and Docker on Mac OS 2017-03-21T00:00:00+00:00 Recently I tried to install Oracle XE into a docker container on Mac OS Sierra using the official docker file</a> from Oracle. I had no luck, because the installation failed with this error message:</p> This system does not meet the minimum requirements for swap space. Based on the amount of physical memory available on the system, Oracle Database 11g Express Edition requires 2048 MB of swap space. This system has 0 MB of swap space. Configure more swap space on the system and retry the installation.</p> </blockquote> The problem is, that you can’t do anything for this on a Mac - the system manages the swap files by itself and if you need for an example 64 GB you will get it. My new standard MacBook has 8 GB RAM and there is currently no reason for a swap file.</p> So, what can we do? After a short search I found this blog post</a>, which describes how to alter the pre-installation script of the RPM installation file with the help of the repmrebuild</code> command. Unfortunately this command is not available on Mac OS. I found only rpmbuild as part of the rpm installation with Homebrew</a>.</p> Luckily I found another way to modify a rpm package under Mac OS: There is a Ruby based tool to build packages for multiple platforms called fpm</a>. So, lets start:</p> If you don’t have Homebrew</a> on your Mac, install this first:</p> /usr/bin/ruby</span> -e</span> \</span></span> "</span>$(</span>curl</span> -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install</span>)</span>"</span></span></code></pre> Install rpm:</p> brew</span> install rpm</span></span></code></pre> Install tar:</p> brew</span> install gnu-tar</span></span></code></pre> Install fpm:</p> gem</span> install --no-ri --no-rdoc fpm</span></span></code></pre> Unzip the downloaded Linux XE installation archive and go into the directory Disk1:</p> unzip</span> oracle-xe-11.2.0-1.0.x86_64.rpm.zip</span> &&</span> \</span></span> cd</span> Disk1</span></span></code></pre> Write rpm scripts to text file:</p> rpm</span> \</span></span> --scripts</span> \</span></span> -qp oracle-xe-11.2.0-1.0.x86_64.rpm</span> \</span></span> ></span> scripts.txt</span></span></code></pre> Find this loop (from line 197 until line 212 as of this writing) and comment out or delete it:</p> # check and disallow install, if swap space is less than Min( 2047, 2 RAM)`</span></span> if</span> [</span> $swapspace</span> -lt</span> $requiredswapspace</span> ]</span></span> then</span></span> if</span> [</span> "</span>$requiredswapspace</span>"</span> =</span> "2047"</span> ];</span></span> then</span></span> requiredswapspace</span>=</span>2048</span></span> fi</span></span> echo</span></span> echo</span> "This system does not meet the minimum requirements for swap space. Based on</span></span> the amount of physical memory available on the system, Oracle Database 11g</span></span> Express Edition requires </span>$requiredswapspace</span> MB of swap space. This system has </span>$swapspace</span> MB</span></span> of swap space. Configure more swap space on the system and retry the</span></span> installation."</span></span> echo</span></span> exit</span> 1</span></span> fi</span></span></code></pre> Find this line (starting on line 328 as of this writing) and delete the line and everything behind until the file end:</p> postinstall</span> scriptlet</span> (using</span> /bin/sh</span>):</span></span></code></pre> Delete the very first line containing this code:</p> preinstall</span> scriptlet</span> (using</span> /bin/sh</span>):</span></span></code></pre> Save the file and bring it back into the rpm with this command (will take some time):</p> fpm</span> \</span></span> -s rpm</span> \</span></span> -t rpm</span> \</span></span> -f --before-install scripts.txt</span> \</span></span> ./oracle-xe-11.2.0-1.0.x86_64.rpm</span></span></code></pre> Go out of Disk1 and update the zip archive, because the Oracle provided build script needs it compressed (will take some time):</p> cd</span> ..</span> &&</span> \</span></span> zip</span> -r oracle-xe-11.2.0-1.0.x86_64.rpm.zip Disk1</span></span></code></pre> When you now do the Docker build you will face another error message:</p> package oracle-xe-11.2.0-1.0.x86_64 is intended for a different operating system</p> </blockquote> To fix this is fairly easy. Open the Dockerfile.xe, go to line 61 (as of this writing) and change this code:</p> rpm</span> -i Disk1/</span></span>.rpm</span> &&</span> \</span></span></code></pre> to this one:</p> rpm</span> -i --ignoreos Disk1/</span></span>.rpm</span> &&</span> \</span></span></code></pre> When you now start the docker build you should be successful (will take some time):</p> sudo</span> ./buildDockerImage.sh -v</span> 11.2.0.2</span> -x -i</span></span></code></pre> The last step is to run the container - on the first startup the database is created (will take some time, align the name to your needs):</p> docker</span> run</span> \</span></span> --name orclapex51</span> \</span></span> --shm-size=1g</span> \</span></span> -p 1521:1521</span> \</span></span> -p 8080:8080</span> \</span></span> oracle/database:11.2.0.2-xe</span></span></code></pre> If you plan to use APEX without any dedicated webserver (maybe because you are the only user and it is only a small dev instance) you should pimp the EPG a little bit:</p> alter system set</span> shared_servers</span>=</span>10</span> scope</span>=</span>both</span>;</span></span> alter system set</span> max_shared_servers</span>=</span>20</span> scope</span>=</span>both</span>;</span></span> alter system set</span> dispatchers</span>=</span>'(PROTOCOL=TCP) (SERVICE=xeXDB) (DISPATCHERS=2)'</span> scope</span>=</span>both</span>;</span></span></code></pre> I use currently a Node.js based proxy server running directly on my Mac OS, because I need a webserver there anyway. The nice thing is, that the EPG don’t need to serve the images and I also don’t need to install the images into the EPG, when I upgrade to a newer APEX version. This saves me time and the EPG runs faster because of less number of requests.</p> Happy installing :-)</p> Ottmar</p> APEX Plugin dhtmlxGantt 2017-03-14T00:00:00+00:00 </p> Some weeks ago I was asked, if I can create a plugin for the Gantt chart library from dhtmlx.com</a>. I had a look at the library and I thought that it should be possible.</p> Because all my plugin developments are free time projects it took a while. Now the first public version is available. It is based on the GPLv2 version of the library, which has a reduced set of functionality. If you need advanced features like milestones, auto scheduling or calculating the critical path you can buy a Pro version of the library. I have no relation to the company behind dhtmlx.com, so please do not complain. As I told before I was asked for this plugin (which will be used in a Pro version there…) and I like the visualization from DHTMLX.</p> You can find the sources and more informations on GitHub</a> and a online demo on apex.oracle.com</a>.</p> Maybe the plugin is helpful for someone else…</p> Happy project planning :-)</p> Ottmar</p> DokuWiki Plugin Reveal.js 2017-03-05T00:00:00+00:00 DokuWiki</a> is very good for creating documentation in a team. It is PHP based, needs less resources and is running without a database only on files on nearly every webserver. If your infrastructure is crashed for whatever reason and you have at least access to the files (on the webserver or in a backup) you can read your documentation with a simple text editor, because DokuWiki’s syntax is designed to be readable as possible without any rendering to HTML.</p> For the APEX connect 2015 I started to use Markdown and Pandoc</a> to create HTML based slides. Pandoc supports different HTML slideshow tools - two of them are also available as plugins in DokuWiki: S5 and Reveal.js</a>. Reveal.js is the more modern library and I used this one together with Pandoc.</p> For the DOAG conference last year I tried to use Reveals.js as a DokuWiki Plugin - I wanted to have at the same time a normal wiki page and the possibility to render this page as a slideshow. Unfortunately the existing plugin was only a base implementation and many features from Reveal.js were not available. I decided to contribute to the existing plugin. The list of my improvements is long - I had fun and learned a lot. The plugin can be found here on the DokuWiki homepage</a>.</p> Happy presenting :-)</p> Ottmar</p> PS: In the meantime I switched complete to Markdown and Jekyll</a>, a static site generator - but this is another story for another post …</p> Markdown Reporter 2017-01-23T00:00:00+00:00 On the APEX connect 2016 in Berlin I talked about my APEX Markdown plugin and fine art printing with the help of Pandoc and LaTeX. When I was preparing my slides and documenting a SQL query in a Markdown code block I asked myself:</p> What, if the database is taking this Markdown code block and runs the query inside to convert it into data?</p> And what, if I take this Markdown text with the data in the code blocks and post it to a small webservice which converts the data blocks into charts and the whole document into HTML or a LaTeX based PDF?</p> The idea to Markdown Reporter was born.</p> I talked about this project on the DOAG conference in Nuremberg last November. Here are the slides</a> (in german, sorry).</p> If you want to know how it looks like - here is a PDF demo report</a></p> The implementation has three main components:</p> An Oracle PL/SQL Package for preprocessing your data and communicating with the printserver</li> A Node.js based printserver, which is in fact a web based remote shell for the format converter Pandoc</a></li> A Pandoc filter written in Python, which intercepts the converting process and generates the charts based on the preprocessed CSV data with the help of Pythons matplotlib</a></li> </ul> The HTML and PDF output is complete vector based and all used tools are open source.</p> You can find the sources and more informations on GitHub</a>.</p> Happy reporting :-)</p> Ottmar</p> Table API Generator 2016-06-10T00:00:00+00:00 Last year I needed for a project a simple table API, which is able to generate a generic change log - something like “show me for table xyz which column was changed when and what was the old value and what is the new one”. Only changed values should be stored and all tables should log into the same generic logging table. Sounds like we need a generator for this.</p> Unfortunately there was no time in the project to create such a generator from the ground. Luckily André, a colleague of me, had started to implement such an API generator in his free time and I had only to implement the generic change log functionality - also in my free time. Our API generator open source project was born :-)</p> Later on I needed a function to get the id from an record by the unique columns - next feature for the API generator: generate a (overloaded) function for each unique constraint on a table. Now it is a simple function call to get the id:</p> your_var :</span>=</span> your_table_name_api.get_pk_by_unique_cols(</span></span> p_column_name_1 </span>=></span> 'xyz'</span>,</span></span> p_column_name_2 </span>=></span> 123</span>,</span></span> ...);</span></span></code></pre> The API is also handling the case, when no data is found and everything without the need to look in the dictionary for the unique columns of that table or to write boilerplate code and possibly forget one column.</p> The next thing for APEX was “For tabular forms you can’t use APIs”. We came up with the idea to generate a view for each table with an instead of trigger for inserts, updates and deletions. The trigger is simply calling the API. Now you can use APEX own “automatic row processing” and the generic logging together.</p> In my holiday this spring I stumbled over a tweet from Phillip Salvisberg about his oddgen project</a>, a SQL Developer extension to invoke dictionary-driven code generators. I was thrilled, we can have a SQL Developer integration by writing a small PL/SQL wrapper for oddgen. I finished the integration in my holiday, I could not wait.</p> Over the time we found out, that the generation of the thin API wrappers for our tables saves us time and makes our code more stable and readable because of the saved boilerplate code. Business logic packages becomes invalid after table changes and API regeneration. This is good, because you can see, which API calls are now invalid. If you have simple insert or update statements you might oversee some places, where the statement is still valid for example with an added table column, but in fact does the wrong insert or update.</p> And the last thing, what is now possible: Security - separate the data from the user interface. Simply grant execute rights on the API and business logic packages to the UI schema and disable the deletion of rows in your API packages when you don’t need it. No user will be able to drop or truncate your tables, because there are no tables in the UI schema.</p> The generated API packages are really thin - nothing special at all. Nevertheless they are powerful time savers.</p> You can find the sources and more informations on GitHub</a>.</p> Happy coding :-)</p> Ottmar</p> Markdown for Oracle APEX 2016-01-01T00:00:00+00:00 Some years ago I implemented a task board in APEX. For commenting I used the stackexchange</a> markdown implementation</a> (old Google code repo</a>), which is a pure JavaScript converter and editor and based on showdown.js</a>.</p> Some months ago I had to implement an application, which should be able to have multiple editors on one page and the editors should be able to support a read only mode. Furthermore there were hard limits for the amount of text for each editor. Both things which are not so easy to implement with the rich text editor in APEX - no really read only mode and the amount of text is also a problem because of the HTML overhead.</p> It was time to rethink my old implementation and to build a plugin, which creates for each text area with the class markdown an own editor. All other non input elements with a class markdown should simply rendered to HTML, which is then also the solution for the read only mode of the editors. APEX standard text area items supports a read only mode and the class markdown from the text area is also rendered by the APEX engine for this mode :-)</p> The nice side effect is, that I don’t had to reinvent the wheel by creating an own item type - I could reuse the standard APEX text area item and things like the character counter working out of the box.</p> You can find the sources and more informations on GitHub</a> and a demo app on apex.oracle.com</a>.</p> Happy coding :-)</p> Ottmar</p> Why a subdomain for an APEX development system is a bad idea 2015-12-12T00:00:00+00:00 Do you find dev.apex.mycompany.tld a nice, rememberable address? Unfortunately this will not work - at least when you use Internet Explorer (which is the standard browser in most companies) and you use the same cookie name in some or all applications to share the session across multiple applications.</em></p> What is the problem on a subdomain?</h2> Internet Explorer including version 11 has a really bad cookie implementation. If you login to your productive APEX instance under apex.mycompany.tld and then to your development instance under dev.apex.mycompany.tld, IE sends the session cookie from your productive instance to your development instance. You will not be able to login - the server is generating a new session each time because of the invalid session cookie.</p> It comes to a nightmare if you configured your systems for automatic login with the windows login credentials as described in this document</a> from Niels de Bruijn. The automatic login leads to hundreds new sessions in a few seconds - depending on the speed of your development system.</p> Not to mention that Chrome and Firefox are working well…</p> More details about IE’s cookie desaster can be found on blogs.msdn.com</a> - the relevant question for this blog entry is number three.</p> Happy APEXing :-)</p> Ottmar</p> D3.js Force-Directed Network Chart 2015-02-20T00:00:00+00:00 </p> My first Oracle APEX plugin and open source project :-)</em></p> Everything started some months ago when I stumbled over this fascinating force-directed graph</a>. I immediately had the idea to use this technology to visualize data models in APEX. My experience is, that direct after I documented a data model, this documentation is out of date. I wanted to have a live visualization of the data models. After the first implementation it turned out, that the resulting graph is also useful for other purposes and I decided to build up an APEX plugin for easier integration.</p> If you use the bare JavaScript files you can run it on every HTML page:</p> <</span>div</span> id</span>=</span>"example"</span>></</span>div</span>></span><!--the graph container--></span></span> <</span>link</span> href</span>=</span>"/d3.js/d3-force-3.0.0.css"</span> rel</span>=</span>"stylesheet"</span> type</span>=</span>"text/css"</span>></span></span> <</span>script</span> src</span>=</span>"/d3.js/ResizeObserver-1.5.0.min.js"</span>></span><</span>/</span>script</span>></span></span> <</span>script</span> src</span>=</span>"/d3.js/d3-3.5.6.min.js"</span>></span><</span>/</span>script</span>></span></span> <</span>script</span> src</span>=</span>"/d3.js/d3-force-3.0.0.min.js"</span>></span><</span>/</span>script</span>></span></span> <</span>script</span>></span></span> window</span>.</span>onload</span> =</span> function</span>() {</span></span> window</span>.</span>example</span> =</span> netGobrechtsD3Force</span>(</span>'example'</span>)</span></span> .</span>height</span>(</span>400</span>)</span></span> .</span>width</span>(</span>800</span>)</span></span> .</span>useDomParentWidth</span>(</span>true</span>)</span> //for responsive layout</span></span> //.zoomMode(true)</span></span> .</span>lassoMode</span>(</span>true</span>)</span></span> .</span>wrapLabels</span>(</span>true</span>)</span></span> .</span>showBorder</span>(</span>false</span>)</span></span> .</span>debug</span>(</span>true</span>)</span> //to enable the customization wizard</span></span> .</span>render</span>()</span>;</span> //sample data is provided when called without data</span></span> //also see https://ogobrecht.github.io/d3-force-apex-plugin/tutorial-1-getting-started.html</span></span> }</span></span> <</span>/</span>script</span>></span></span></code></pre> I found myself often fiddling around with the parameters of the physical simulation - change config, reload page, not amused, change config…</p> Really annoying - even I know the implementation behind. So, the question was: What can I do for an easy configuration. My answer was after a while of thinking and trying out: A customization wizard with a live preview of the changes. The link to enter the wizard is shown when the debug mode is switched on or in APEX when the developer bar is shown - try it out on the getting started page of the API docs</a>.</p> You can find the sources and more informations on GitHub</a> and a demo app on apex.oracle.com</a>.</p> Happy networking :-)</p> Ottmar</p> Double click on APEX report row opens first link target 2015-01-14T00:00:00+00:00 Do you like scanning a report for certain information and then moving the mouse to the first column to click the small link for details or a edit form? Would it be nice to double click anywhere on the row to open the first link?</em></p> With a little help of a dynamic action on page zero you have the solution for all your interactive and standard reports in your application. The only additional to do is, to find out the right selectors for your reports. Here some usual ones:</p> //APEX 5:</span></span> table</span>.</span>a</span>-</span>IRR</span>-</span>table tr</span>,</span> table</span>.</span>uReportStandard tr</span></span> </span> //APEX 4:</span></span> table</span>.</span>apexir_WORKSHEET_DATA tr</span>,</span> table</span>.</span>uReportStandard tr</span></span></code></pre> I give you no guarantee that this will also work with your theme. Please ask your browsers developer tools and inspect your report layouts ;-)</p> Create a dynamic action for a double click on a jQuery selector (see examples above) and set the event scope to dynamic.</p> The dynamic action needs to execute JavaScript:</p> $</span>(</span>this</span>.</span>triggeringElement)</span>.</span>find</span>(</span>'td:first>a'</span>)[</span>0</span>]</span>.</span>click</span>()</span>;</span></span></code></pre> Happy coding :-)</p> Ottmar</p>

Yet another Oracle DB logging tool: Console

2021-10-04T00:00:00+00:00

It looks like it is a hobby of PL/SQL developers to develop their own logging tool. There are already some free tools on the market and probably many that have never been released (DOAG talk by Sabine Heimsath on the topic</a>):

Logger</a></li>
PIT</a></li>
Instrumentation for PLSQL</a></li>
Log4plsql</a></li>
ILO</a></li>
BMC_DEBUG</a></li>
…</li> </ul>
One reason seems to be that everyone has different ideas or needs. For me, I wanted a logging tool that is very easy to install and works even if you are not allowed to create a context in the database and also have no special read permissions for administrative views like v$session. You only need the rights to create tables and packages and optionally a cleanup job - pretty standard. Nevertheless, it is possible to move individual users/sessions to a higher log level for debugging purposes. As this is solved via client identifier, this will also work in an environment without fixed session ID like e.g. APEX. If no client identifier is set in an environment, then Console simply assigns one itself. Console reads its configuration from a table with only one line supported by the result cache. This ensures a resource-saving execution. Also the check whether a log message is really written to the log table based on the current log level is highly optimized to keep the overhead as low as possible in production environments.
A single install script</h2>
For Console, all scripts are merged into a single install script. Since SQLcl can also load scripts from the Internet, you could install the tool in a minute without downloading it first: Invoke SQLcl, log into the desired schema, and call @https://raw.githubusercontent.com/ogobrecht/console/main/install/create_console_objects.sql</code>. A few seconds later you can start logging. If you want to install it in APEX and only have browser access to your development environment, the single install script in SQL Workshop is also very helpful and Console installs quickly.
Production safe without further configuration</h2>Console logs only errors (level 1) by default. This means that you are on the safe side on production systems without any further configuration. But if you want to enable other levels like warning (2), info (3), debug (4) or trace (5) on a development system and don’t want to do this for each session individually, you can set it globally: exec console.conf(p_level => 3);</code>. More about this in the package description for the procedure console.conf</a>. To set a session (client identifier) to a higher log level use the procedure console.init</a>. If you omit the parameter p_client_identifier</code>, then the client identifier of your own session is automatically taken - here is an example: -- Dive into your own session with the default log level of 3 (info) and the -- default duration of 60 (minutes). exec console.init; -- With level 4 (debug) for the next 15 minutes. exec console.init(4, 15); -- Using a constant for the level exec console.init(console.c_level_debug, 90); -- Debug an APEX session... begin console.init( p_client_identifier => 'OGOBRECHT:8805903776765', p_level => console.c_level_debug, p_duration => 15); -- there are more parameters availabe... end; /</code></pre> Now the question arises, how to get the client identifier of a foreign session without read access to adminstrative views like e.g. v$session. A variant would be to write this for example in the frontend of an application with sys_context('USERENV', 'CLIENT_IDENTIFIER')</code> - e.g. in the footer or on a help page. If you want to return to the normal mode of global settings for all sessions, you can do so with console.exit</a>. You should not use the procedures console.conf</code>, init</code> and exit</code> in your business logic - they are only used for managing session settings and for debugging purposes and should therefore only be used interactively or in SQL scripts. Method names based on JavaScript Console</h2> Console uses as many method names from the JavaScript Console as possible - so switching between backend code and frontend code shouldn’t be that hard as far as method names are concerned. Whether the methods really write anything to the log table CONSOLE_LOGS depends on the active log level - so first these: Level 1: Error</li> Level 2: Warning</li> Level 3: Info</li> Level 4: Debug (instead of verbose in the JavaScript Console)</li> Level 5: Trace (not available in the JavaScript Console)</li> </ul> The main instrumentation methods: console.error_save_stack (more on that in a moment):</li> console.error (level error)</li> console.warn (level warning)</li> console.info & log (level info)</li> console.debug (level debug)</li> console.trace (level trace)</li> console.count & count_reset</li> console.count_current & count_end (level info)</li> console.count_current & count_end (function overloads, independent of log level)</li> console.time & time_reset</li> console.time_current & time_end (level info)</li> console.time_current & time_end (function overloads, independent of log level)</li> console.table# (level info)</li> console.assert & assertf</li> console.format</li> console.add_param</li> </ul> More in the API overview</a>. Reduced amount of log entries through saved call stack</h2> Console uses the capabilities of the utl_call_stack</code> package to reduce the number of log entries to a possible minimum. Who doesn’t know the problem: In case of an error, a log entry is created in each sub-function to record as many details as possible. In the end you have to watch how the log table gets cluttered and you try to find out from the many log entries where exactly the error occurred. It would be helpful to see the method names in the error backtrace - but the database only writes the package names and the line number in the backtrace. To work around this problem Console offers the possibility instead of writing an error in the submethods into the log table, to save the call stack with the call console.error_save_stack</code> until finally in the outermost main method console.error</code> is called, which then enters the error including saved call stack into the log table. For clarification, here is a script with a test package: set define off set feedback off set serveroutput on set linesize 120 set pagesize 40 column call_stack heading "Call Stack" format a120 whenever sqlerror exit sql.sqlcode rollback prompt TEST ERROR_SAVE_STACK prompt - compile package spec create or replace package some_api is procedure do_stuff; end; / prompt - compile package body create or replace package body some_api is ------------------------------------------------------------------------------ procedure do_stuff is -------------------------------------- procedure sub1 is -------------------------------------- procedure sub2 is -------------------------------------- procedure sub3 is begin console.assert(1 = 2, 'Demo'); exception --sub3 when others then console.error_save_stack; raise; end; -------------------------------------- begin sub3; exception --sub2 when others then console.error_save_stack; raise; end; -------------------------------------- begin sub2; exception --sub1 when others then console.error_save_stack; raise no_data_found; end; -------------------------------------- begin sub1; exception --do_stuff when others then console.error; raise; end; ------------------------------------------------------------------------------ end; / prompt - call the package begin some_api.do_stuff; exception when others then null; --> I know, I know, never do that without a final raise... --> But we want only test our logging without killing the script run... end; / prompt - FINISHED, selecting now the call stack from the last log entry... select call_stack from console_logs order by log_id desc fetch first row only;</code></pre> Here is the output of the above script - the Saved Error Stack section is Console’s special feature, the other three stack and trace sections are the database’s standards: TEST ERROR_SAVE_STACK - compile package spec - compile package body - call the package - FINISHED, selecting now the call stack from the last log entry... Call Stack ------------------------------------------------------------------------------------------------------------------------ #### Saved Error Stack - PLAYGROUND.SOME_API.DO_STUFF.SUB1.SUB2.SUB3, line 14 (line 11, ORA-20777 Assertion failed: Demo) - PLAYGROUND.SOME_API.DO_STUFF.SUB1.SUB2, line 22 (line 19) - PLAYGROUND.SOME_API.DO_STUFF.SUB1, line 30 (line 27) - PLAYGROUND.SOME_API.DO_STUFF, line 38 (line 35, ORA-01403 no data found) #### Call Stack - PLAYGROUND.SOME_API.DO_STUFF, line 38 - __anonymous_block, line 2 #### Error Stack - ORA-01403 no data found - ORA-06512 at "PLAYGROUND.SOME_API", line 31 - ORA-20777 Assertion failed: Test assertion with line break. - ORA-06512 at "PLAYGROUND.SOME_API", line 23 - ORA-06512 at "PLAYGROUND.SOME_API", line 15 - ORA-06512 at "PLAYGROUND.CONSOLE", line 750 - ORA-06512 at "PLAYGROUND.SOME_API", line 11 - ORA-06512 at "PLAYGROUND.SOME_API", line 19 - ORA-06512 at "PLAYGROUND.SOME_API", line 27 #### Error Backtrace - PLAYGROUND.SOME_API, line 31 - PLAYGROUND.SOME_API, line 23 - PLAYGROUND.SOME_API, line 15 - PLAYGROUND.CONSOLE, line 750 - PLAYGROUND.SOME_API, line 11 - PLAYGROUND.SOME_API, line 19 - PLAYGROUND.SOME_API, line 27 - PLAYGROUND.SOME_API, line 35</code></pre> If you don’t use console.error_save_stack</code> but always console.error</code>, then you get at least the last three sections in the log - and without extra work in the code. You just have to remember console.error</code>. Simple logging of method parameters</h2> Console also provides a simple way to log method parameters. Here is a sample procedure with parameters of all supported types: --create demo procedure create or replace procedure demo_proc( p_01 varchar2, p_02 number, p_03 date, p_04 timestamp, p_05 timestamp with time zone, p_06 timestamp with local time zone, p_07 interval year to month, p_08 interval day to second, p_09 boolean, p_10 clob, p_11 xmltype) is begin raise_application_error(-20999, 'Test Error.'); exception when others then console.add_param('p_01', p_01); console.add_param('p_02', p_02); console.add_param('p_03', p_03); console.add_param('p_04', p_04); console.add_param('p_05', p_05); console.add_param('p_06', p_06); console.add_param('p_07', p_07); console.add_param('p_08', p_08); console.add_param('p_09', p_09); console.add_param('p_10', p_10); console.add_param('p_11', p_11); console.error('Ooops, something went wrong'); raise; end demo_proc; /</code></pre> In the exception handling you can see nicely that you always call the same procedure console.add_param</code> and pass the name and the value. The parameters are cached in an array in the console package (shortened to a maximum of 2000 characters) and taken over with the next call of a log method (error, warn, info, log, debug or trace). If you don’t want the parameters to be truncated, you are free to write the parameter directly into the log message - this is of type clob and is therefore not subject to any size restrictions. Here an example call of the above procedure: begin demo_proc( p_01 => 'test vc2', p_02 => 1.23, p_03 => sysdate, p_04 => systimestamp, p_05 => systimestamp, p_06 => localtimestamp, p_07 => interval '4-2' year to month, p_08 => interval '7 6:12:42.123' day to second, p_09 => true, p_10 => to_clob('test clob'), p_11 => xmltype('<test_xml/>')); end; /</code></pre> This call then writes the following log message - see the MESSAGE column: Markdown format for automatically determined metadata</h2> The attentive reader might not have missed that the above log message is formatted in Markdown. So, if you like, you can have your report rendered accordingly in HTML (e.g. in APEX) - however, it is well readable in text form, too. The Markdown table format as in the example above is also used by Console for logging other metadata such as APEX environment, CGI environment, user environment and console environment. All of these environments can be turned on for each individual log call. They are then appended to the log message like the parameters. Here exemplarily the signature of the error procedure: procedure error( p_message in clob default null, -- The log message itself p_permanent in boolean default false, -- Should the log entry be permanent (not deleted by purge methods) p_call_stack in boolean default true, -- Include call stack p_apex_env in boolean default false, -- Include APEX environment p_cgi_env in boolean default false, -- Include CGI environment p_console_env in boolean default false, -- Include Console environment p_user_env in boolean default false, -- Include user environment p_user_agent in varchar2 default null, -- User agent of browser or other client technology p_user_scope in varchar2 default null, -- Override PL/SQL scope p_user_error_code in integer default null, -- Override PL/SQL error code p_user_call_stack in varchar2 default null); -- Override PL/SQL call stack</code></pre>Extensible logs through overloaded log methods</h2> The error procedure has an overload in the form of a function that returns the log ID. Thus one can extend the logging also with own data in own tables e.g. for a downstream release process in case of specific errors. This is also the purpose of the p_permanent</code> parameter, which ensures that the cleanup job or the console.purge</code> and console.purge_all</code> procedures do not delete the correspondingly marked log entries and that these are permanently available. All other log methods (warn, info, log, debug, trace) are implemented in the same way and with the same parameters - but some have different default values. For the error method the call stack is written, for the trace method all four environment details. The parameters p_user_agent</code>, p_user_scope</code>, p_user_error_code</code> and p_user_call_stack</code> are intended to be able to capture external log events as well and to overwrite the automatically determined values of the PL/SQL environment. As an example an external load process in a data warehouse or error messages from the JavaScript frontend of an application can be mentioned. With a little imagination, everyone will come up with their own use cases here… Measure time and count things</h2> Measuring execution times and counting things are very common functions. Console can help here to not have to create too many helper variables and keep the code short and sweet. For this purpose it offers the procedures time</code>, count</code> and other helpers. You can easily run several timers or counters in parallel - they are each identified by an optional label. If you omit the label, then internally the label is default</code>. The best way to illustrate this is a bit of sample code: begin --basic usage console.time; sys.dbms_session.sleep(0.1); console.time_end; -- without optional label and message console.time('myTimer'); sys.dbms_session.sleep(0.1); console.time_current('myTimer'); -- without optional message sys.dbms_session.sleep(0.1); console.time_current('myTimer', 'end of step two'); sys.dbms_session.sleep(0.1); console.time_end('myTimer', 'end of step three'); end; /</code></pre> This will result in the following log messages in the console_logs table if the current log level is 3 (info) or higher: default: 00:00:00.102508</li> myTimer: 00:00:00.108048</li> myTimer: 00:00:00.212045 - end of step two</li> myTimer: 00:00:00.316084 - end of step three</li> </ul> But sometimes you don’t want to have prefabricated log entries or you need the elapsed time to write them in scripts to the server output. For this purpose the procedures time_current</code> and time_end</code> used in the code are also available as overloaded functions. With these you can do what you want: set serveroutput on begin console.time; --console.print is an alias for sys.dbms_output.put_line console.print('Processing step one...'); sys.dbms_session.sleep(0.1); console.print('Elapsed: ' || console.time_current); console.print('Processing step two...'); sys.dbms_session.sleep(0.1); console.print('Elapsed: ' || console.time_current); console.print('Processing step three...'); sys.dbms_session.sleep(0.1); console.print('Elapsed: ' || console.time_end); end; /</code></pre> This then leads to something like this server output: Processing step one... Elapsed: 00:00:00.105398 Processing step two... Elapsed: 00:00:00.209267 Processing step three... Elapsed: 00:00:00.313301</code></pre> For counting things or processes there are the same procedures and functions as for time measurements - only with the prefix count</code> instead of time</code> (we use the variant without label in the example): set serveroutput on begin console.print('Counting nonsense...'); for i in 1 .. 1000 loop if mod(i, 3) = 0 then console.count; end if; end loop; console.print('Current value: ' || console.count_current ); console.count_reset; for i in 1 .. 10 loop console.count; end loop; console.print('Final value: ' || console.count_end ); end; /</code></pre> This then results in a server output as follows: Counting nonsense... Current value: 333 Final value: 10</code></pre> The count_reset</code> procedure used in the example also exists for timers and is called time_reset</code> there. The respective *_end</code> methods delete the entry from the list of timers/counters managed in the console package. Assert, format and other helpers</h2> There are other auxiliary methods that make the developer’s life more pleasant. First of all, the assert procedure should be mentioned. Everyone should know this recurring pattern: if not x < y then raise_application_error(-20999, 'X should be less than Y'); else -- your code here end if;</code></pre> It gets especially ugly when you have to check multiple conditions. The whole thing can also be formulated as a one-liner: console.assert(x < y, 'x should be less then y');</code></pre> If you want to enrich texts with dynamic information, then you are quickly in concatenation hell: console.assert( x < y, 'x should be less then y (x=' || to_char(x) || ', y=' || to_char(y) || ')');</code></pre> Here the format</code> function can help you: console.assert( x < y, console.format( 'x should be less then y (x=%s, y=%s)', to_char(x), to_char(y)));</code></pre> format</code> accepts up to 10 parameters and works according to the following rules: replace all occurrences of %0</code> … %9</code> with the corresponding parameters p0</code> … p9</code></li> replace %n</code> with new lines (line feed character)</li> replace all occurrences of %s</code> in positional order with the corresponding parameters with sys.utl_lms.format_message - see also the Oracle docs</a></li> </ol> I personally am very reluctant to type sys.dbms_output.put_line</code>. Console does write to a log table with all the log methods, but actually the word console suggests that we can also write directly to the server output, right? For that we have the procedure print</code>: console.print('A message');</code></pre> Then there are the short forms of console.print(console.format(...))</code> and console.assert([boolean expression], console.format(...))</code>: console.printf( 'A dynamic message with a %n second line of text: %s', my_var); console.assertf( x < y, 'x should be less then y (x=%s, y=%s)', to_char(x), to_char(y));</code></pre> Also often needed is a fast, cached clob concatenation - that’s where Console can help with clob_append</a> & clob_flush_cache</a>. More can be found in the API overview</a>. Displaying the package status of Console in a session</h2> If you are interested in what is configured in the current session of Console, you can look at it with a pipelined table function or make it available in your application with a report: select * from table(console.status);</code></pre>ATTRIBUTE</th> VALUE</th></tr></thead> c_version</td> 1.0.0</td></tr> localtimestamp</td> 2021-10-03 13:58:00</td></tr> sysdate</td> 2021-10-03 11:58:00</td></tr> g_conf_check_sysdate</td> 2021-10-03 11:58:10</td></tr> g_conf_exit_sysdate</td> 2021-10-03 12:58:00</td></tr> g_conf_client_identifier</td> {o,o} 11ECD3500002</td></tr> g_conf_level</td> 5</td></tr> level_name(g_conf_level)</td> trace</td></tr> g_conf_check_interval</td> 10</td></tr> g_conf_enable_ascii_art</td> true</td></tr> g_conf_call_stack</td> false</td></tr> g_conf_user_env</td> false</td></tr> g_conf_apex_env</td> false</td></tr> g_conf_cgi_env</td> false</td></tr> g_conf_console_env</td> false</td></tr> g_counters.count</td> 1</td></tr> g_timers.count</td> 2</td></tr> g_saved_stack.count</td> 0</td></tr> g_prev_error_msg</td> </td></tr> </tbody></table> APEX Error Handling Function</h2> For APEX, Console comes with a so-called “Error Handling Function</a>” which can enter errors within the APEX runtime environment into the log table. If you want to use this, you have to enter this function in your application in the “Application Builder” under “Edit Application Properties > Error Handling > Error Handling Function”: console.apex_error_handling</code>. APEX error handling with ASCII art enabled</figcaption> </figure> The Error Handling Function logs the technical error in the CONSOLE_LOGS table and writes a friendly message to the end user. It uses the APEX Text Message feature for the user-friendly messages in case of constraint violations, as described in this video</a> by Anton and Neelesh from Insum, which in turn is based on an idea by Roel Hartman in this blog post</a>. The APEX community rocks…. APEX Plug-in for capturing frontend errors</h2> Furthermore, there is an APEX Dynamic Action Plug-in, which enters JavaScript errors in the user’s browser via AJAX call into the log table. This way you also get to know if the frontend is not running smoothly for the users… You have to make sure that Console is either installed in the parsing schema of the application or a synonym named CONSOLE has been created in the parsing schema which points to the package CONSOLE. Then you can install the plug-in in install/apex_plugin.sql</code> and create a Dynamic Action for the whole application on page zero: Event: Page Load</li> Action: Oracle Instrumentation Console [Plug-in]</li> No further customization required (only a JavaScript file will be loaded)</li> </ul> For those interested in what the plug-in does: sources/apex_plugin_console.js</code>. This is currently a minimal implementation and may be improved in the future. Sources of inspiration</h2> I am not alone in the world and without the others not much goes in software development. Here are a few links to sites or projects that inspired me on the way to my own logging tool: JavaScript Console API</a></li> Logger</a></li> Instrumentation for PLSQL</a></li> PIT</a></li> Oracle Magazine - Sophisticated Call Stack Analysis</a></li> </ul> Special thanks to Dietmar Aust</a>, who gave me time and ideas with a discussion session at an early stage. Project homepage</h2> Console is hosted on GitHub</a>. Happy logging and debugging :-) Ottmar Create fast insert scripts 2021-01-31T00:00:00+00:00 A few weeks ago I released a new version of my PLEX project</a>. I improved the ability to export table data as insert scripts. Previously, PLEX was only able to export CSV data. The reason was that I originally developed PLEX to speed up the implementation of version control in Oracle DB projects, and the ability to export CSV data was intended to track changes in master data tables. It turned out that a colleague of mine wanted PLEX to export insert scripts for all data in all tables for the initial deployment of a new application. I implemented an initial version, but was not happy with the runtime of the insert scripts. I now hear some people arguing that there are better tools than insert scripts for importing larger amounts of data - but sometimes this is the only accepted way because only SQL*Plus is available to run the deployment scripts…. Then I stumbled across an older post by Connor McDonald on creating fast insert scripts</a>. I have integrated his performance hacks into PLEX. You can use them as follows (see last parameter): declare l_zip_file blob; begin l_zip_file := plex.to_zip(plex.backapp( p_app_id => 100, -- parameter only available when APEX is installed p_include_ords_modules => true, -- parameter only available when ORDS is installed p_include_object_ddl => true, p_include_data => true, p_data_as_of_minutes_ago => 0, -- Read consistent data with the resulting timestamp (SCN). Defaults to 0. p_data_max_rows => 10000, -- Maximum number of rows per table. Defaults to 1000. p_data_table_name_like => 'CAT\_%,USERS,ROLES,RIGHTS', p_data_table_name_not_like => '%\_HIST,LOGGER%', p_data_format => 'insert:10')); -- Do something with the zip file. -- Your code here... end; /</code></pre> The p_data_format</code> parameter is new and can contain a comma-separated list of formats - currently the CSV and INSERT formats are supported. An example: csv,insert</code> exports for each table a CSV file and a SQL file with insert statements. For insert you can also specify the number of rows per insert all</code> statement (default value is 20) - example: csv,insert:10</code> or insert:5</code>. As you can see from the p_data_table_name_not_like => '%\_HIST,LOGGER%'</code> parameter, PLEX also accepts lists with like filters. In our example %\_HIST,LOGGER%</code> is translated to: where ... and ( table_name not like '%\_HIST' escape '\' and table_name not like 'LOGGER%' escape '\' and ... )</code></pre> For the parameter p_data_table_name_like => 'CAT\_%,USERS,ROLES,RIGHTS'</code> the tables are filtered as follows: where ... and ( table_name like 'CAT\_%' escape '\' or table_name like 'USERS' escape '\' or ... )</code></pre> There is a good reason why the default p_data_max_rows</code> parameter is set to 1000</code> - if you don’t specify table name filters, all tables will be exported… The resulting insert scripts look like this: -- Script generated by PLEX version 2.4.2 - more infos here: https://github.com/ogobrecht/plex -- Performance Hacks by Connor McDonald: https://connor-mcdonald.com/2019/05/17/hacking-together-faster-inserts/ -- For strange line end replacements a big thank to SQL*Plus: https://support.oracle.com/epmos/faces/DocumentDisplay?id=2377701.1 (SQL Failed With ORA-1756 In Sqlplus But Works In SQL Developer) prompt - insert 847 rows into USERS (exported 2021-01-31 14:24:30.703234000 +00:00) set define off feedback off sqlblanklines on alter session set cursor_sharing = force; alter session set nls_numeric_characters = '.,'; alter session set nls_date_format = 'yyyy-mm-dd hh24:mi:ss'; alter session set nls_timestamp_format = 'yyyy-mm-dd hh24:mi:ssxff'; alter session set nls_timestamp_tz_format = 'yyyy-mm-dd hh24:mi:ssxff tzr'; begin insert all into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (1,'Ylcmsajbil','Fojkjryntnixzfh','qvspjgvwmtbi@ghovilkddx.mly') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (2,'Axify','Cofjlkwzxytdih','ajgttnqlds@minokpyfo.gu') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (3,'Ckyqqvuqrkuktb','Igacqwp','qpygabuhbrs@nsjxpgjlle.ze') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (4,'Noythhl','Gausfu','ngmgsbr@duyxqzn.hmyo') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (5,'Kpewbrecnfzsi','Nwbsnjh','xwlhcfaxko@uhqsibdojjp.hsm') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (6,'Qlzyizkl','Gwnaojlvyud','kzndqj@nsosenf.fm') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (7,'Xpepttcemrd','Ktaqqdnqyfvc','uhbnzezvz@buiptt.lkrm') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (8,'Ybyinr','Vngairocujhy','igvfzoegbh@hsepkqiwbst.evs') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (9,'Ctjixltj','Yvsiei','ozpspssyw@vooiyfuf.xeh') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (10,'Lttzwsavnozxu','Kcyjalvzrl','yvwowaqrpku@dyapdumb.fvi') select * from dual; /* S N I P */ insert all into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (841,'Gpkabykoveq','Gljhlrijqop','imnhrheyr@ypccyiu.ah') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (842,'Mwrfazphbvmekpw','Kxirzfth','fxoatt@frlbwbn.tf') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (843,'Hzxomvkliaxl','Mstdrrmgfmsy','gpeidglzwfa@hwyumsansy.fet') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (844,'Vwvwezfsd','Xtfouojiymtlu','zgsdtowsvt@ywfngnijgts.ozd') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (845,'Eljntjfkxx','Sifgii','gksggat@ubfmmdopqy.ly') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (846,'Ewvyjeudjb','Anihlpdgeg','gietuk@ezciwejuedy.nuf') into USERS(U_ID,U_FIRST_NAME,U_LAST_NAME,U_EMAIL) values (847,'Wscdamsgssmouf','Omtaofvlrjs','jrwyzftmbmo@gjylnuez.esq') select * from dual; end; / commit; alter session set cursor_sharing = exact; set define on</code></pre> PLEX saves the NLS settings of the current session and makes sure that the insert scripts run with the same settings - this means that you should not get any number or date conversion errors. The performance hacks from Connor are alter session set cursor_sharing = force;</code> and insert all</code> - thanks for pointing that out Connor :-) PLEX has many parameters to configure the export - the focus of PLEX is to quickly introduce version control to existing projects (see also this article about PLEX in general</a>). If you want to use PLEX only for exporting table data, then you should create a wrapper function for it that suits your personal needs - here is an example: create or replace function to_insert( p_table_name varchar2, p_max_rows integer default 1000, p_batch_size integer default 20) return clob is l_file_collection plex.tab_export_files; begin l_file_collection := plex.backapp( p_include_data => true, p_data_max_rows => p_max_rows, p_data_table_name_like => p_table_name, p_data_format => 'insert:' || to_char(p_batch_size), --overwrite defaults to save work for plex: p_include_templates => false, p_include_runtime_log => false, p_include_error_log => false); return l_file_collection(1).contents; end; /</code></pre> Now you can always do a quick export like this: select to_insert('USERS', 10000) from dual;</code> Happy exporting and scripting :-) Ottmar New Version of Oracle Table API Generator Available 2020-12-26T00:00:00+00:00 Short before Christmas André and I released version 0.6.0 of our table API generator</a>. The last release was exactly two years ago - time flies. This blog post highlights some of the changes - the complete changelog is available on GitHub</a>. Bulk Processing</h2> When you split your app over multiple schemas for higher data security you might need at some point the possibility to bulk process your data. Imagine the following, common schema setup (also see this nice video by Connor McDonald</a>, 20 minutes): Data Schema: Holds your tables and provides table APIs for the other schemas, which have only select rights on the tables (or views) and execute rights on the table APIs</li> API Schema: Holds your business logic and provides business APIs to the app parsing schema</li> UI Schema: Has no own objects, only connect rights and is used as the parsing schema to run the application</li> </ul> This setup follows the principle of the least privilege. When you need to bulk process your data in the business logic (API schema), then you have a problem without bulk processing facilities in your table APIs. You need to implement business logic in the data schema or you grant update and/or delete rights to the API schema. The first solution is against the separation of concerns and the second one lowers your data security and maybe prevents you from not having triggers (more about this later in the audit columns section). That is why we have now also bulk processing in the core of our generated table APIs. Here is how you can use it in your business logic: /* EXAMPLE INSERT To keep the memory usage low we break our inserts into a bulk size of 1000 rows. You should adapt the bulk size to your use case. */ declare l_rows_tab data_schema.your_table_api.t_rows_tab; l_number_bulks integer := 100; l_bulk_size integer := 1000; begin l_rows_tab := data_schema.your_table_api.t_rows_tab(); l_rows_tab.extend(l_bulk_size); <<number_bulks>> for z in 1 .. l_number_bulks loop <<bulk_size>> for i in 1 .. l_bulk_size loop l_rows_tab(i).column_1 := your_logic_here; l_rows_tab(i).column_2 := your_logic_here; end loop bulk_size; data_schema.your_table_api.create_rows(l_rows_tab); commit; end loop number_bulks; end; /</code></pre>/* EXAMPLE UPDATE */ declare l_rows_tab data_schema.your_table_api.t_rows_tab; l_ref_cursor data_schema.your_table_api.t_strong_ref_cursor; begin -- optionally set bulk limit, default is 1000 -- data_schema.your_table_api.set_bulk_limit(500); open l_ref_cursor for select * from app_users where your_conditions_here; <<outer_bulk>> loop l_rows_tab := data_schema.your_table_api.read_rows(l_ref_cursor); <<inner_data>> for i in 1 .. l_rows_tab.count loop --do your business logic here l_rows_tab(i).column_1 := your_logic_here; end loop inner_data; data_schema.your_table_api.update_rows(l_rows_tab); commit; exit when data_schema.your_table_api.bulk_is_complete; end loop outer_bulk; close l_ref_cursor; end; /</code></pre> The bulk deletion works the same way as the bulk update - the difference is that you do not modify your data before calling the delete_rows</code> method. For more examples see the docs: Basic usage</a></li> Performance tests</a></li> </ul> Support for Audit Columns</h2> Keeping the bulk processing in mind for performance reasons it is a bad idea to have triggers only to manage audit columns in a table. If you have your tables in an own schema and the business logic and the app cannot directly write to the tables then it is easy to prevent triggers. Therefore the table APIs need to support that and hides the audit columns for insert or update operations. This is what our generator now supports. You can map your specific audit column names to the four often used names created</code>, created_by</code>, updated</code> and updated_by</code> by providing a mapping string to the new parameter p_audit_column_mappings</code> - an example: -- TABLE DEFINITION -- create table app_users ( au_id integer generated always as identity, au_first_name varchar2(15 char) , au_last_name varchar2(15 char) , au_email varchar2(30 char) not null , au_active_yn varchar2(1 char) default 'Y' not null , -- This is only for demo purposes. In reality we expect -- more unified names and types for audit columns. au_created_on date not null , au_created_by char(15 char) not null , au_updated_at timestamp not null , au_updated_by varchar2(15 char) not null , -- primary key (au_id), unique (au_email), check (au_active_yn in ('Y', 'N')) ); begin om_tapigen.compile_api( p_table_name => 'APP_USERS', p_audit_column_mappings => 'created=#PREFIX#_CREATED_ON, created_by=#PREFIX#_CREATED_BY, updated=#PREFIX#_UPDATED_AT, updated_by=#PREFIX#_UPDATED_BY' ); end; /</code></pre> The #PREFIX#</code> in our example is optional (determined automatically by the generator) and replaced with the found column_prefix au</code>. Doing it this way you can provide always the same audit column mapping to all of your tables. I know some people hate column prefixes - but if you design your data models this way the generator can save you the work to align the mapping for each table. Support for a Row Version Column</h2> The same reasons with triggers for the audit columns are valid for the support of a row version column. This can also be handled by our generator: begin om_tapigen.compile_api( p_table_name => 'YOUR_TABLE', p_row_version_column_mapping => '#PREFIX#_VERSION_ID=tag_global_version_sequence.nextval' ); end; /</code></pre> You can also see here the before mentioned #PREFIX#</code> substitution. The difference is that you define here only your column and for the column the SQL expression which should be used for the row version column - in this case, a sequence. What we cannot do here is to add always 1 to the previous value like it is possible in a trigger - the table API has no idea what the previous value was. So, we are a little bit limited to a SQL expression but the point is, you can let the generated API manage the row version column without having a trigger. Tenant Support</h2> This is the third new column mapping definition. For all people who have not an enterprise version with the VPD (virtual private database) functionality you can define a tenant expression for one of your columns: begin om_tapigen.compile_api( p_table_name => 'YOUR_TABLE', p_tenant_column_mapping => '#PREFIX#_TENANT_ID=to_number(sys_context(''my_sec_ctx'',''my_tenant_id''))' ); end; /</code></pre> For more information see the parameter docs for p_tenant_column_mapping</a>. More</h2> A new optional 1:1 view can be generated (new parameter p_enable_one_to_one_view</code>)</li> All generated object names can now be customized (new parameters p_dml_view_name</code>, p_dml_view_trigger_name</code> and p_one_to_one_view_name</code>)</li> Double quoting of table and column names in the generated code can now be configured (new parameter p_double_quote_names</code>, default true)</li> A new helper function get_default_row</code> will be generated, when parameter p_enable_column_defaults</code> is set to true (to support the row based create_row function)</li> We have now unit tests driven by utPLSQL (it will be a permanent task to improve the tests with every new feature or bugfix)</li> </ul> Breaking Changes</h2> Where there is light, there is also shadow. All the new features like bulk processing and audit column support make it hard, if not impossible, to generate a generic change log during the row processing. Also the check, if we have a modification and omit an update if not necessary is hard to implement and not good for the performance with the evolved generator. So we skipped the mentioned functionality. As far as we know not many people use the generic changelog. If you do, you have to stick with the previous generator version or you have to find an alternative way to do it in the future. A good way seems to be the use of the Flashback Data Archive as it is performant and available in all database editions for free - and you can load existing data into the archive (see Tims article below): Blog post by Connor McDonald: Triggers can provide auditing information, but there’s a future in flashback</a></li> Blog post by Tim Hall: Flashback Data Archive (FDA) Enhancements in Oracle Database 12c Release 1 (12.1)</a></li> Video by Connor McDonald: Flashback Data Archive</a> (14 minutes)</li> Video APEX Connect 2020 by Connor McDonald: Flashback – so much more</a> (47 minutes)</li> Video AskTOM Office Hours by Chris Saxon: The Incredible Flashback Data Archive</a> (53 minutes)</li> </ul> And the last Breaking change is, that you need at least Oracle DB 12.1 or higher. That should (hopefully) no problem for the most projects. Conclusion</h2> A big THANK YOU to all contributors - the current version of the generator would not be possible without your feedback and help. Be sure to test the new generator before migrating to it. We use it already in some projects and it should be pretty stable. Nevertheless, with so many changed things we cannot give you a guarantee for every edge case - by using it you accept the MIT license. Please check out the project page at GitHub</a> and open an issue</a> if you encounter a problem. A happy new year and happy coding :-) Ottmar How to migrate Oracle Cloud wallet into existing auto_login_local wallet 2020-08-03T00:00:00+00:00 I started to store user credentials in wallet files for higher security in SQL scripts. If you do so you should also secure the access to your wallet files on the operating system level. This is especially true for auto-login wallets from the Oracle Cloud. To increase the security a bit further you can restrict the auto-login to the local user who created the wallet. To achieve this you need to create an own wallet with orapki</code> instead of mkstore</code>: orapki wallet create \ -wallet "path/to/my/wallet" \ -pwd myPassword \ -auto_login_local</code></pre> You can then add new alias/user/password entries with mkstore (you need to add each alias to your tnsnames.ora): mkstore \ -wrl "path/to/my/wallet" \ -createCredential myAlias myUser myPassword</code></pre> If you use only the instant client and cannot find mkstore and orapki, then have a look at my previous post</a> on how to simulate these tools. The Migration</h2> I assume you have set an environment variable called TNS_ADMIN pointing to your TNS directory. In this directory, you have your wallet files and also your existing tnsnames.ora and sqlnet.ora to successfully connect to your local Oracle instances. Now you need to extract the downloaded cloud wallet - for security reasons it makes sense to create a temporary subdirectory in your TNS directory and extract the cloud wallet there. You need to migrate the entries from tnsnames.ora</code> and sqlnet.ora</code> into your own TNS files and align the path to your wallet directory. Additionally, you should also copy over the file ojdbc.properties</code> if you do not have it already and you plan to use SQLcl or SQL Developer with your wallet. My resulting sqlnet.ora: WALLET_LOCATION = (SOURCE = (METHOD = file) (METHOD_DATA = (DIRECTORY = "C:\\path\\to\\my\\wallet"))) SSL_SERVER_DN_MATCH = yes sqlnet.wallet_override = true</code></pre> My resulting tnsnames.ora: myCloudDbName_high, myAlias = (description = (... myCloudDbName_medium = (description = (... myCloudDbName_low = (description = (... myCloudDbName_tp = (description = (... myCloudDbName_tpurgent = (description = (... myLocalDbName, myOtherAlias = (description = (... ...</code></pre> That was the easy part, but how you migrate the needed private key and all certificates? It took me some time to figure this out because most people seem to use the cloud wallet as it is. This is understandable since SQL Developer can direct use the zipped version of the wallet. But my target is not to manage the passwords in multiple places - the databases, multiple wallets, SQL Developer and possibly other tools. I want to manage the passwords only in the databases and in one wallet and use this wallet with all the tools - in my case SQL*Plus, SQLcl and SQL Developer. First I tried to export the private key and all certificates from the cloud wallet and then import it into my wallet. I was not successful with this mostly because of the private key and unknown alias names - maybe I was too tired from all the try and error things on how to achieve this. I came finally up with the idea to use the cloud wallet Java KeyStore files and convert them into my wallet with the orapki method jks_to_pkcs12</code>. Migrate private key and certificate from cloud wallet: orapki wallet jks_to_pkcs12 \ -wallet "path/to/my/wallet" \ -pwd myPassword \ -keystore "path/to/cloud/wallet/keystore.jks" \ -jkspwd myCloudWalletPassword</code></pre> Migrate trusted certificates from cloud wallet: orapki wallet jks_to_pkcs12 \ -wallet "path/to/my/wallet" \ -pwd myPassword \ -keystore "path/to/cloud/wallet/truststore.jks" \ -jkspwd myCloudWalletPassword</code></pre> Finally, list the wallet content to see the migrated entries: orapki wallet display -wallet . -pwd myPassword # or orapki wallet display -wallet . -pwd myPassword -complete</code></pre> The final file list in my TNS directory (without the Java KeyStore files from the cloud wallet): cwallet.sso cwallet.sso.lck ewallet.p12 ewallet.p12.lck ojdbc.properties sqlnet.ora tnsnames.ora</code></pre> I am now able to connect to my cloud database with the desired tools without providing user/password details (tested version): SQL*Plus (19.6): sqlplus /@myAlias</code></li> SQLcl (20.2): sql /@myAlias</code></li> SQL Developer (20.2): authentication type “OS”, connection type “Custom JDBC” with URL jdbc:oracle:thin:/@myAlias</code></li> </ul> If you forgot which aliases you have configured in your wallet: mkstore -wrl "path/to/my/wallet" -listCredential</code></pre> Hope this helps someone else. Happy developing and scripting :-) Ottmar How to use mkstore and orapki with Oracle Instant Client 2020-07-29T00:00:00+00:00 For some years I use only the Oracle Instant Client</a>. I had too often trouble with the full client installation on my corporate PCs under Windows, especially in the old Windows 32/64 bit times. The instant client can be used by download and extract one basic zip file and adding an entry to your path environment variable. If you want to use SQL*Plus and other tools you need to download two additional files - no drama, also without admin rights. When I started to store the user details in wallet files I was wondering why Oracle does not deliver the mkstore and orapki tools with the instant client. I searched for a workaround and found some posts about it - including installing the full client or other software to get the tools. But I do not want to install additional software only for two small command-line tools. I found the final hint in an old post from Andriy Dmytrenko</a>. Seems that the two utilities are only shell scripts that are calling some Java libraries. The question was from which location I should take the needed libraries. Since I also have SQLcl</a> on my PC I found the libs there. Below my final shell scripts for Windows - put them somewhere and have the path pointing to the directory where they are stored. If you need them for Linux you will find a way to do it by yourself: mkstore.bat: @echo off rem inspiration: https://andriydmytrenko.wordpress.com/2013/07/01/using-the-secure-external-password-store-with-instant-client/ setlocal rem get the command line arguments set args= :loop if !%1==! goto :done set args=%args% %1 shift goto :loop :done rem set classpath for mkstore - align this to your local SQLcl installation set sqlcl=c:\path\to\sqlcl\lib set classpath=%sqlcl%\oraclepki.jar set classpath=%classpath%;%sqlcl%\osdt_core.jar set classpath=%classpath%;%sqlcl%\osdt_cert.jar rem simulate mkstore command java -classpath %classpath% oracle.security.pki.OracleSecretStoreTextUI %args% endlocal</code></pre> orapki.bat: @echo off rem inspiration: https://andriydmytrenko.wordpress.com/2013/07/01/using-the-secure-external-password-store-with-instant-client/ setlocal rem get the command line arguments set args= :loop if !%1==! goto :done set args=%args% %1 shift goto :loop :done rem set classpath for orapki - align this to your local SQLcl installation set sqlcl=c:\path\to\sqlcl\lib set classpath=%sqlcl%\oraclepki.jar set classpath=%classpath%;%sqlcl%\osdt_core.jar set classpath=%classpath%;%sqlcl%\osdt_cert.jar rem simulate orapki command java -classpath %classpath% oracle.security.pki.textui.OraclePKITextUI %args% endlocal</code></pre> If you need a starter on using mkstore and orapki then Tim Hall from Oracle Base has covered you with an article about the secure external password store</a>. Happy password storing :-) Ottmar Sample Data Sets for Oracle 2020-05-25T00:00:00+00:00 Sometimes you need only some small tables and some rows of data to play around with, to test things out. I think most of you immediately would say EMP, DEPT… Sure, but a bit more data would be nice. Ok, OE and HR schema… Sure, but I have no SYS rights on the database in my current project to install the schemas, can I have it in my current dev schema?… Sure, but then I have to fiddle a bit with the scripts to make it run in my dev schema, I need it fast, no time at all… Mmhhh, sounds this familiar to you? Maybe I can help… The sample data sets from Oracle were normally created for a dedicated schema. I changed that and omitted all the schema creation parts of the scripts. Instead, all objects from a data set get a prefix - an example: All objects from the customer orders schema (one of the newer data sets from Oracle) are prefixed with CO_. This allows me to use all data sets in parallel in a single schema and easily identify the data set the object belongs to. All table scripts are created with identity columns and the simplest possible options for easy readability - no fancy constraint and index names. Constraints and indexes will be renamed with global helper scripts after the table creation. Only real indexes are coded - foreign key indexes are generated with a global helper script too. For each data set, you have always the same three scripts to handle the installation/data refresh/uninstallation. List of Data Sets, Copyrights</h2> EMP & DEPT (2 tabs, 18 rows), source</a>, Copyright Oracle (MIT license</a>)</li> Order Entry & Human Resources (15 tabs, 3.002 rows), source oe</a>, source hr</a>, Copyright Oracle (MIT license</a>)</li> Customer Orders (5 tabs, 6.325 rows), source</a>, Copyright Oracle (MIT license</a>)</li> Sakila DVD Rental Store (16 tabs, 46.273 rows), source</a>, original source</a>, Copyright MySQL AB documentation team, DB Software Laboratory, Lukas Eder (BSD-3-Clause license</a>)</li> </ul> For more info about the Oracle sample schemas see the docs</a>. List of Global Helper Scripts</h2> Disable/enable all foreign key constraints</li> Create missing foreign key indexes</li> Unify constraint names</li> Unify index names</li> Sync sequence values to data</li> </ul> You can find the project on GitHub</a>. Hope this helps someone else… Happy data modeling and testing :-) Ottmar P.S. For those of you who are working with Oracle APEX: Have a look under SQL Workshop > Utilities > Sample Datasets Why I Like SQL*Plus for Deployment Scripts 2020-01-02T00:00:00+00:00 The following simplified example deployment master script is given: set define off verify off feedback off whenever sqlerror exit sql.sqlcode rollback spool deployment.log prompt prompt Example Deployment Master Script prompt ======================================== prompt Step 1: Dummy for demonstration -- normally we call here some other scripts like -- @other-script.sql prompt Step 2: Error test in anonymous block begin raise_application_error( -20000, 'Error test for SQL*Plus <> SQLcl comparison' ); end; / prompt ======================================== prompt Deployment Done :-) prompt spool off</code></pre> When we call this in SQL*Plus (version 19.5), we get the following output (in the console and also in deployment.log) with an indication of what went wrong with our deployment: Example Deployment Master Script ======================================== Step 1: Dummy for demonstration Step 2: Error test in anonymous block begin * ERROR at line 1: ORA-20000: Error test for SQL*Plus <> SQLcl comparison ORA-06512: at line 2</code></pre> Ok, this error message is stupid, but usually, I get some useful information here to fix my problem in the deployment script - especially when the problematic script is a bit longer… When we call this in SQLcl (latest version 19.4 on Windows) we get the following output (in the console and also the deployment.log): Example Deployment Master Script ======================================== Step 1: Dummy for demonstration Step 2: Error test in anonymous block</code></pre> Mhhh… What was going on here? Why was my script terminating in step 2? That’s one reason why I like SQL*Plus a bit more than SQLcl for deployment scripts. The fast start time of SQL*Plus is another reason. Nevertheless, SQLcl is a cool command-line tool. Happy scripting :-) Ottmar PS: There is a small Twitter discussion</a> around this… UPDATE January 3, 2020</h2> Jeff Smith has confirmed that this bug was fixed in SQLcl version 19.4</a>. I was using 19.4 - but on Windows. It turns out that the bug is still there - only on Windows, not Mac, not Linux… So, if you are facing this problem on Windows then you have to wait for a future release or set as a workaround the feedback to on: set define off verify off feedback on <snip></code></pre> Here the output on my Windows 10 laptop: Demo Deployment Master Script ======================================== Step 1: Dummy for demonstration Step 2: Begin error test in anonymous block Rollback Error starting at line : 10 File @ C:\temp\deployment.sql In Command - begin raise_application_error(-20000, 'error test for SQL*Plus <> SQLcl comparison'); end; Error report - ORA-20000: error test for SQL*Plus <> SQLcl comparison ORA-06512: at line 2 20000. 00000 - "%s" *Cause: The stored procedure 'raise_application_error' was called which causes this error to be generated. *Action: Correct the problem as described in the error message or contact the application administrator or DBA for more information.</code></pre> If you are on Mac OS or Linux then you have to update to the latest SQLcl version 19.4 or use also the described workaround. UPDATE July 20, 2020</h2> The bug still exists in SQLcl 20.2 and SQL Developer 20.2 under Windows - we have to wait a bit longer… UPDATE January 31, 2021</h2> For me it is working now with SQL Developer 20.4.0 under Windows and also with SQLcl 20.4.1, when I have the environment variable path pointing to the JDK delivered within SQL Developer (...\sqldeveloper\jdk\jre\bin\</code>). I had the path before pointing to another standalone Java 11 distribution and it was not working there… Again: happy scripting :-) Ottmar Download BLOBs with SQL*Plus 2020-01-01T00:00:00+00:00 Have you ever tried to spool/download BLOBS with SQL*Plus? Some months ago I tried to find a way to download a collection of CLOBs or even better a zipped version of the collection as BLOB. I learned that BLOBs are not a valid datatype in SQL*Plus when it comes to the spool command. I found a way by putting the CLOBs into a global temporary table, creating an intermediate SQL script with tons of select and spool commands, and finally spooling the CLOBs one by one by calling the intermediate SQL script. It was working, but far from elegant. Also, when I was in the home office with a less powerful connection, it was slowing down the whole export process significantly. If you are using PLEX (PL/SQL Export Utilities) and this sounds familiar to you - the described method was the default for exporting an APEX app or your schema DDL with the delivered export script templates. If you have no idea what I am talking about: PLEX is a PL/SQL package that is able to export an Oracle APEX app, ORDS modules, all schema objects and table data into a nice directory structure in one go ready to use for version control. Check out the project on GitHub</a> and this blog post on how to get started</a>. But there is a better way: Simply convert the BLOB into a base64 encoded CLOB. I have no idea why this was not my first solution for the PLEX export templates. In the new version 2.1.0 of PLEX the templates are using this approach. Here is a simplified example. File export.sql: set verify off feedback off heading off set trimout on trimspool on pagesize 0 linesize 5000 long 100000000 longchunksize 32767 whenever sqlerror exit sql.sqlcode rollback variable contents clob begin :contents := plex.to_base64(plex.to_zip(plex.backapp( p_app_id => 100, p_include_ords_modules => true, p_include_object_ddl => false, p_include_data => false, p_include_templates => true))); end; / set termout off spool "app_100.zip.base64" print contents spool off set termout on</code></pre> Congratulations - You are the owner of a base64 encoded CLOB on your client disk. And now? Before you can unzip the file you need to decode it - depending on your operating system you can do this with your OS tools: Windows: certutil -decode app_100.zip.base64 app_100.zip</code></li> Mac: base64 -D -i app_100.zip.base64 -o app_100.zip</code></li> Linux: base64 -d app_100.zip.base64 > app_100.zip</code></li> </ul> Here a simple Windows batch file, which calls the SQL script above, decodes the file and deletes the base64 version. File export.bat echo exit | sqlplus -S demo/oracle@localhost:1521/xepdb1 @export.sql certutil -decode app_100.zip.base64 app_100.zip del app_100.zip.base64</code></pre> The described method to download a BLOB is independent of the package PLEX - you only need a small helper function to encode the BLOB. I found an example provided by Tim Hall on his Oracle Base site</a>. Thanks for sharing Tim! CREATE OR REPLACE FUNCTION base64encode(p_blob IN BLOB) RETURN CLOB -- ----------------------------------------------------------------------------------- -- File Name : https://oracle-base.com/dba/miscellaneous/base64encode.sql -- Author : Tim Hall -- Description : Encodes a BLOB into a Base64 CLOB. -- Last Modified: 09/11/2011 -- ----------------------------------------------------------------------------------- IS l_clob CLOB; l_step PLS_INTEGER := 12000; -- make sure you set a multiple of 3 not higher than 24573 BEGIN FOR i IN 0 .. TRUNC((DBMS_LOB.getlength(p_blob) - 1 )/l_step) LOOP l_clob := l_clob || UTL_RAW.cast_to_varchar2(UTL_ENCODE.base64_encode(DBMS_LOB.substr(p_blob, l_step, i * l_step + 1))); END LOOP; RETURN l_clob; END; /</code></pre> This function is included in the newest version of PLEX (named to_base64) - you can download PLEX here</a>. Happy scripting, exporting and version controlling :-) Ottmar Quick start - version control for existing Oracle projects 2019-07-20T00:00:00+00:00 Many Oracle projects still do not use version control. The reasons for this are manifold. Mostly it is assumed that the database is a safe place for the source code. With a working backup this is also correct, but in any case one loses the complete history of changes. Often in running projects there is not enough time to deal with the introduction of source code versioning, because at first sight there is no direct benefit to be seen. Who dares to take a second look under time pressure? This article wants to lower the hurdle for the introduction of a version management a little bit. Note DOAG</h2> This article appeared in DOAG Red Stack magazine 03-2019 and is also available in the german original</a>. Unlike the original, updates may occur here on my blog if relevant things change. For example, since version two of the mentioned PLEX (PL/SQL Export Utilities) project, an APEX installation is no longer a requirement. This means that also pure Oracle database projects can use PLEX - read more in this blog entry</a>. Introduction</h2> The first step is the most important one, as everyone knows who has ever finally set out after much hesitation - whatever it was about. The same is true for source code versioning. Once you’re on your way, you wonder how you got along for so long without it. A versioned source code base increases your comfort zone during development, and the time invested pays off several times over. But how do you take the first step as easily and quickly as possible? How to design the structure of a repository? Workflow change: file-based working</h2> One of the most important questions is: Can I continue to work as before? The answer depends entirely on how one has worked so far. Basically, working with a source code versioning also means a change away from developing in the database. Changing and compiling code directly in the database or changing tables with the tool of one’s choice must evolve to file-based work. This means working with a file in the local source code repository with the tool of choice. The changes are then compiled from the file. For schema changes, scripts must contain the necessary DDL code. Depending on the tools used so far, this is rather a subtle difference when editing logic, but a big difference when changing database objects like tables. If a schema diff tool was previously used to transport changes from the development environment into production, then this might even look like a step backwards at first glance. The point here is that you can only get a usable source code history if you are consistent about working file-based. There can also be an intermediate step towards this approach. One continues to work as before and regularly exports all objects of the schema to the source code management. This way one gets at least a certain ordered transparency of the changes. If then in the course of the project the advantages of this procedure become apparent, one usually still lacks the information, who executed the respective changes. In addition, the transparency gained quickly awakens desires such as “If I have the old code in the repository, then I may try something, I can go back if necessary”. The first hurdle is already taken and the openness to perhaps really get started is there. That something like this can work even better is something that most developers come up with themselves. For example, creating a branch (development branch) in the local repository. This way you can try something out without running the risk of having to discard your changes due to an unexpected bug. You can easily switch to the main (parallel) branch, work on the bugfix, update your branch to the changed main branch and then just keep trying. All this without the danger of losing things or influencing other colleagues. Have I lost anyone now? No problem, just get started - only practice helps here. Tool comparison: DDL export</h2> How do you accomplish the first step? How do you export the schema objects of your application into a well sorted directory structure? Most of the used graphical development tools provide an export function for this purpose. Depending on the tool, this is more or less suitable for building a source code repository. Therefore, let’s take a look at how the most commonly used tools SQL Developer, PL/SQL Developer and Toad can do this. A word beforehand: the intention of the schema export of these tools is actually to bring the objects into another schema, maybe even into another database, and not to create a source code repository. The following questions move us when it comes to building one: Is one script file per object possible?</li> Are subdirectories per object type possible?</li> Are separate files for foreign keys possible? (Useful for simpler master scripts)</li> Can “Object already exist” errors be prevented? (Or in other words: are the scripts restartable?)</li> Can data be exported? (If possible in CSV format for tracking master data changes).</li> Is it possible to export an APEX app? (Possibly also broken down into the individual parts such as pages, shared components, etc.)</li> </ul> Figure 1 shows the result related to the question. Criterion</th> SQL Dev.</th> PL/SQL Dev.</th> Toad</th></tr></thead> File per object</td> Yes</td> Yes</td> Yes</td></tr> Subdir. per type</td> Yes</td> No</td> Yes</td></tr> FK constr. separate</td> Yes</td> No</td> Yes</td></tr> Prevent “object exist”</td> No</td> No</td> No</td></tr> Export data</td> Yes</td> No</td> partly</td></tr> Export APEX app</td> partly</td> No</td> No</td></tr> </tbody></table> Figure 1: Tool comparison DDL export</figcaption> </figure> Notes on SQL Developer: Is the clearest</li> Many formats for data export (also CSV)</li> Extensively configurable</li> Export APEX App only with Commandline version SQLcl</li> </ul> Notes on PL/SQL Developer: Little configurable</li> Disappointing for building a source code repository</li> </ul> Notes on Toad: Two export options (at least). Either subdirectories per object type….</li> … or data</li> </ul> </li> Data only as insert statements</li> Extensively configurable, relatively unclear</li> </ul> None of the tools provides us with a ready-made, well-structured source code repository. You have to do more or less a lot of rework. That doesn’t mean it can’t be done, especially since sorting the objects into a reasonable structure should be a one-time job. Only in the case of exporting the APEX application one would like to have an automatic solution, because this export will have to be done more often in the future. This is in the nature of APEX as a declarative “low code framework”. We develop the APEX application in the browser and the meta-data of the application resides in the APEX repository of the database. An export of the application can also be done with the browser, but in doing so you will only get a single large SQL file with the entire application. Desirable for a source code repository would be an export of the individual parts such as pages, shared components, plugins, etc.. This would then make it possible for the application to track how it has developed and also a search in the repository for e.g. a specific package call makes much more sense with a disassembled APEX application. In the past, you could use the APEX export splitter for this, basically a Java class that comes with every APEX version to this day. The disadvantage is that you have to take the directory structure as created by the splitter or you change it afterwards with local scripts on your PC. However, it would be nicer to be able to adjust the directory structure already during the export in order to be able to store all master scripts in a central folder of the repository. Since APEX version 5.1.4 there is the APEX_EXPORT package for this purpose. With this package you can export either the whole script or the splitted version. The return format in both cases is a collection - each line of the collection contains a path specification for the respective export file and a CLOB field with the content. So you could edit this collection in an export script and adapt the path specifications to your own needs. This is exactly what the tool package PLEX, published as open source by the author of this article, does - the name stands for PL/SQL Export Utilities. Furthermore, PLEX can answer yes to any of the above questions about DDL export - no wonder, it was designed for that. Quickstart: Package PLEX</h2> PLEX aims to be able to select the first version of the repository as a BLOB with a simple query. After saving the BLOB to the local directory system on the PC with the extension .zip you can unpack it and find in the generated directory structure script examples for future semi- or fully automatic exports and deployments via shell command or SQL*Plus. A possible first export could look like shown in Listing 1. with function backapp return blob is begin return plex.to_zip(plex.backapp( p_app_id => 100, /* If null, we simply skip the APEX app export. Parameter only available if APEX installed. */ p_include_object_ddl => true, /* If true, include DDL of current user/schema and all its objects. */ p_include_templates => true, /* If true, include templates for README.md, export and install scripts. */ p_include_runtime_log => true, /* If true, generate file plex_backapp_log.md with runtime statistics. */ p_include_data => true, /* If true, include CSV data of each table. */ p_data_max_rows => 1000, /* Maximum number of rows per table. */ p_data_table_name_like => 'DEMO_PRODUCT_INFO,DEMO_STATES' /* A comma separated list of like expressions to filter the tables. Example: 'EMP%,DEPT%' will be translated to "where ... and (table_name like 'EMP%' escape '\' or table_name like 'DEPT%' escape '\')". */ )); end backapp; select backapp from dual;</code></pre> Listing 1: Possible first export</figcaption> </figure> Since PLEX has boolean parameters, we use an inline function in the with clause. If you use a database version smaller than 12c, you can create an auxiliary function similar to the example. PLEX has some more parameters to configure e.g. the APEX App Export. More details can be found on the official project page at github.com/ogobrecht/plex</a>. Depending on the size of the schema and the APEX app, this initial call can take anywhere from a few seconds to several minutes. This is because in the background for each object the package DBMS_METADATA is used to generate the DDL of the object. If you are interested in runtime information, you will find a log file in the main directory of the export with information about what PLEX did and how long the respective steps took. PLEX itself now exists in a first version - suggestions for improvements or bug reports are welcome and can be reported as issues on the project page. From here on it strongly depends on the needs of the respective project if and how often you run a DDL export. The normal way should be a regular export of the APEX application. Everything else should be processed locally from now on and as a result there should be no need for an export. However, as mentioned at the beginning, there may be situations where you regularly export all schema objects in order to document the application. However, this should only be seen as an intermediate step towards a file-based way of working. Generated code is a special case: So for example using Quick-SQL in APEX or generating table APIs. Here one can consider exporting the generated code with appropriately configured object filters. For APIs you should consider versioning the generator instead of the generated code - but this depends entirely on whether the generated code is still being manually modified or not. Any manually generated code should be versioned. Geschwindigkeit: Immer Skripte</h2> As you can already see from the script examples provided by PLEX, the goal should be to execute any deployment by script. This also includes the consideration to export the object DDL only at the very first time - because what happens e.g. when exporting table DDL? If you have written another column into the table script via Alter Table statement and made sure that it is restartable (see Listing 2), then another DDL export overwrites the Alter statement and our table script simply contains the new column listed in the Create Table statement. However, this is never executed again due to the restartability, since our table already exists. begin for i in ( select 'DEMO_STATES' as object_name from dual minus select object_name from user_objects ) loop execute immediate q'[ ---------------------------------------- create table "demo_states" ( "st" varchar2(30), "state_name" varchar2(30) ) ---------------------------------------- ]'; end loop; end; / begin for i in ( select 'STATE_DESCRIPTION' AS column_name from dual minus select column_name from user_tab_columns where table_name = 'DEMO_STATES' and column_name = 'STATE_DESCRIPTION' ) loop execute immediate q'[ ---------------------------------------- alter table demo_states add ( state_description varchar2(255) ) ---------------------------------------- ]'; end loop; end; /</code></pre> Listing 2: Restartable script</figcaption> </figure> Thus, we now have a source code file that is no longer suitable for deployment. A pity actually, because until just now we could have simply called all object scripts with our master script and only the changes (like our new column) would really have been executed - a simple deployment and a clear history in the repository. This is contrasted with the automatic creation of diff scripts, which are sometimes not restartable and risk data loss even with slightly more complicated schema changes. One cannot avoid checking the generated scripts for correctness and making manual adjustments for any data migrations, even with very expensive tools of this class. No one takes the decision of which way to go away from you. Now we are in the middle of the discussion on how to deploy to the target system. PLEX can’t help here either - except that the package per se tries to extract all necessary object DDLs in a way that they can be restarted. Here we are in the wide field of individual needs of a project. The script examples supplied by PLEX must be adapted to the needs of the respective project in any case. For example, the author almost always distinguishes between scripts for the backend and the frontend - both for export and deployment. Consideration: Directory structure in the repository</h2> How should you structure your repository? Here is a suggestion that has proven to be beneficial for me personally over time. Every project may have different ideas. I use a simple listing here - the first words of each list item represent the directory name, and any explanations are in parentheses after it: app_backend (our schema DDL) constraints (one folder per object type)</li> package_bodies</li> packages</li> ref_constraints</li> sequences</li> tables</li> …</li> </ul> </li> app_data (tracking of master data, optional)</li> app_frontend (our APEX app) pages</li> shared_components</li> …</li> </ul> </li> docs (the documentation)</li> reports</li> scripts (all master scripts combined) logs (export and installation logs) export_app_100_from_MYSCHEMA_at_DEV_20190315_132542</li> install_app_100_into_MYSCHEMA_at_TEST_20190318_083756</li> …</li> </ul> </li> misc (frequently used scripts like recompile schema)</li> 1_export_app_from_DEV.bat (shell script with parameters for the actual SQL export script)</li> 2_install_app_into_TEST.bat (Shell script with parameters for the SQL deployment script)</li> 3_install_app_into_PROD.bat (shell script with parameters for the custom SQL deployment script)</li> export_app_custom_code.sql (export master script with customizations)</li> install_app_custom_code.sql (Deployment Master Script with individual customizations per release)</li> install_backend_generated_by_plex.sql (called from master script)</li> install_frontend_generated_by_apex.sql (called by the master script)</li> </ul> </li> tests (unit and frontend tests)</li> README.md (general information about the project and links to the documentation)</li> </ul> The repository should be logically structured and not contain extremely deep directory structures to make it easier to use. All master scripts are united in one script folder, also the installation script generated by APEX for the frontend. All master scripts should store their respective feedback in corresponding log files in the logs subfolder during use - this makes exports of source code and deployments to target systems traceable. Outlook: CI/CD & Schema Migration</h2> The outlined discussion of deployment already hints at it: Simply setting up a source code repository and you’re done is not the solution. That was just the first step to warm up. Once you have structured everything and made it executable via script, you can automate the deployment in the second step. The content of this discussion is far beyond the scope of this article, which focuses on the first step. Therefore, the author suggests everyone not to stop after this first step and to consider further steps. Martin Fowler, for example, has written a very good basic article on the subject - he considers every change to the database as a migration. For further study, the following entry points should therefore be mentioned: Antti Kirmanen: Git vs. Subversion (SVN): Welches Versionskontrollsystem sollten Sie nutzen?</a></li> Martin Fowler: Evolutionary Database Design</a></li> Samuel Nitsche: There is no clean (database) development without Version Control</a></li> “One does not simply update a database” – migration based database development</a></li> </ul> </li> Blain Carter: Tips to help PL/SQL developers get started with CI/CD</a></li> CI/CD for Database Developers – Export Database Objects into Version Control</a></li> </ul> </li> Jeff Smith: 19.X SQLcl Teaser: LIQUIBASE</a></li> </ul> Happy version controlling :-) Ottmar New Major Version of PL/SQL Export Utilities Available 2019-06-26T00:00:00+00:00 PLEX is a PL/SQL package with export utilities - see this post</a> what it can do for you. In the past it was dependent on APEX for two reasons: The APEX_EXPORT package to allow exporting an APEX app and the APEX_ZIP package for compressing the resulting file collection to a zip file. After the first public release of PLEX, I got a comment like “Nice tool, but I don’t use APEX”. I thought I should make PLEX independent from APEX to allow at least the export of all the schema DDL and also the table data. Some weeks ago Jürgen Schuster contacted me to try out PLEX for an switch from Subversion to Git for one of his customer projects. It turns out that PLEX was not able to export the schema DDL because of errors from DBMS_METADATA regarding many Java classes. And the second issue was that one of their machines had no APEX installed. This was the final trigger for me to rework the implementation of PLEX. PLEX is now independent from an APEX installation. Thanks to Anton Scheffer for his as_zip package</a>. PLEX reuses the methods for compressing files and uses conditional compilation to include/exclude functionality depending on APEX. Because PLEX cannot rely on APEX anymore, the return type for the methods backapp and queries_to_csv has changed from apex_t_export_files</code> to plex.tab_export_files</code>. This was unavoidable and if you have existing scripts and you want to update to the new PLEX version, you have to change the variables based on that return type. A simple search and replace will normally do the trick. This breaking change in the PLEX API is also the reason for a new major version. The other improvement is better error handling and logging. Per default you will find two files in an export collection: plex_runtime_log and plex_error_log. The former is for an overview what happened and how long it took and the latter is for detailed error messages. If you encounter errors and you think that should not happen and is a problem of PLEX, then please open an issue on GitHub</a>. I was able to export the schema APEX_19_0100 with around 5000 objects on my dev machine… Project on Github</a></li> Download latest version</a></li> </ul> Happy exporting :-) Ottmar New Version of D3 Force Network Chart Available 2019-06-09T00:00:00+00:00 First of all: Thanks to the community! From time to time someone has new ideas or questions about the D3 network chart. Some time ago I thought my D3 implementation is more or less feature complete. But in the last weeks I worked again on the code to bring in new feature wishes from the community. The most visible one in the new version 3.1 is the ability to add labels to links - see the links between Allen, Blake and Ward in the screenshot above. The labels are positioned in a way that they are always readable - this is the reason why they appear sometimes above and sometimes below a link line (they are rotated, if needed). The nice thing is that you can also read both labels when you have links in both directions between two nodes. The other benefit of having labels on links is that they extend the area for the link click and hover events. This improves the usabiltiy in cases where the links should show a tooltip when they are hovered or some action should be executed when they are clicked. This is normally not so easy because the links are so narrow. There are also new helper methods and one new option - for the full change log see the project on GitHub</a> and for online demos either the standalone version in the docs</a> or the APEX plug-in version on apex.oracle.com</a>. Happy networking :-) Ottmar PLOC - Convert PL/SQL Code into Markdown Docs 2019-01-10T00:00:00+00:00 Some months ago I had to write a README file with Markdown syntax for a PL/SQL open source project. I changed function and procedure parameters of the package often during the implementation. Therefore, maintaining the documentation was no fun in the beginning. Maybe you know this from your own (payed) projects ;-) Normally in the business world I see minimalistic comments in the code and a dedicated documentation independent of the code. As you can imagine, the latter is sometimes - let’s say “out of date”, if it exists at all … I don’t want to discuss the reasons for this here - you all know several ones. And I also do not have a general solution for code documentation. But I have some ideas what is important for me: The base of a good PL/SQL code documentation is a good code structure No single functions and procedures - packages with meaningful names are the way to go</li> </ul> </li> As I have the source code under version control in local files on my computer I need a command line tool for the doc generation - there is no need for a database connection The CLI should be able to process multiple files (packages)</li> For my small open source project I have only one package and I want to define a document name different then the package name, so I need a CLI parameter for that</li> </ul> </li> Comments should be written directly after the package/function/procedure signature (in the package spec, optionally it should also work for triggers and types) I want to use Markdown for the comments - this allows me to write also bigger text parts and code examples</li> </ul> </li> I don’t want to describe all parameters in a JavaDoc manner PL/SQL is strong typed, everything I need to know is visible in the signature of a function or procedure</li> Replicating the parameters for documentation is error prone and does not follow the DRY principles</li> With meaningful parameter names you need often no or only short comments - these can be placed directly after the parameters as single line comments</li> Long descriptions can be written in the Markdown main comment</li> </ul> </li> The package/function/procedure signature should be automatically included in the output as a code block for reference</li> Output should also be Markdown - this can be easily read by humans and further processed by machines Only documented functions/procedures should be included in the output to keep the focus on the important stuff</li> I want to have an automatic table of contents in each generated document</li> With the help of Pandoc</a> many formats like Word, RTF, PDF or HTML can be generated (for GitHub I only need Markdown)</li> </ul> </li> </ul> There are already several open source projects available for PL/SQL code documentation generation: PLDoc</a></li> OraDoclet</a></li> SchemaSpy</a></li> Natural Docs</a></li> PL/SQL to Markdown Documenter</a></li> </ul> Only the last one is able to generate Markdown docs, but the input has to be in JavaDoc. All the others can produce HTML output and often some other formats like RTF, Word or PDF. Natural Docs has a Markdown like simple text syntax, but also no Markdown output. Some of the tools can automatically document a whole schema - that is not the focus of PLOC. Most comparable is the intention of PL/SQL to Markdown Documenter. So I had to do it by myself. I called it PLOC (PLDoc was already used, so I skipped the D…) and it is implemented in JavaScript as a npm package and currently used by me for the following two projects: PL/SQL Export Utilities</a> and Oracle Table API Generator</a>. It is very lightweight - the main code without the CLI has currently less characters as the Markdown source of this blog post. Maybe it is also useful for you. If not, have a look at the other mentioned projects. More infos about PLOC</a> itself can be found on the project page</a>. Happy coding and documenting :-) Ottmar Table API Generator v0.5.0 Available 2018-12-26T00:00:00+00:00 Short before Christmas André and I released version 0.5.0 of our PL/SQL table API generator</a>. It took a long time to come to this point. One of the reasons behind was the fact that we had to rework some internals to support feature wishes from the community. Here the most important features of the new version: Support for multicolumn primary keys</li> Support for 12c long identifier names</li> More control over the API features and name (10 new parameters)</li> Insert methods: Can have the dictionary column defaults</li> Can return the whole row instead of the primary key</li> </ul> </li> You can exclude columns (e.g. audit columns) for inserts and updates</li> Improved template engine: Generated code better readable because of formatted parameter lists</li> Tables with many columns no longer breaking character limits</li> </ul> </li> Special methods for testing and dummy data generation</li> Also see the full changelog on the project site</a></li> </ul> Happy new year and happy coding :-) Ottmar New Major Version of D3 Force Network Chart Available 2018-12-02T00:00:00+00:00 Since APEX version 5 was published with the Universal Theme, I was thinking that I needed something for better responsiveness of the graph. Although it was possible to configure the graph to use the DOM parent width, the responsiveness was not so good because of situations where the window size does not change (we use the window resize event to trigger the change of the width) but the available space for the graph changes. How can the graph react to this? The problem is that a network chart is not a simple image which can be resized according to the available space. The graph is an SVG with multiple layers to handle the zoom and pan, the lasso, a legend that needs to be in a fixed size… As you can see, there must be a procedural solution. And for this we need a trigger, which always fires when the (possible) size of the graphs parent DOM element changes. Fortunately, there is a W3C draft for a resize observer</a> and a polyfill</a> available. To put the things together, I had to make breaking changes on the API (the reason for a new major version): I made the API zoom methods independent of the option zoomMode</code> and set the default to true for the following options: zoomToFitOnForceEnd</code> (was false in the past)</li> zoomToFitOnResize</code> (new option)</li> keepAspectRatioOnResize</code> (new option)</li> </ul> When setting the option useDomParentWidth</code> to true together with the previous mentioned defaults, you can achieve a responsiveness like with images set to width 100%. Try it out by resizing your browser window. For the full change log see the project on GitHub</a> and for a demo the app on apex.oracle.com</a> (which currently does not use the universal theme, new demo app is work in progress…). Happy networking :-) Ottmar PLEX - PL/SQL Export Utilities 2018-08-26T00:00:00+00:00 PLEX is a standalone PL/SQL package with export utilities. It was created to be able to quickstart version control for existing (APEX) apps. It currently has two main functions called backapp</code> and queries_to_csv</code>. queries_to_csv</code> is used by BackApp as a helper function, but its functionality is also useful as a standalone. This post is all about BackApp, which has the following features: Export the app definition of an APEX app (splitted files and optional single SQL file)</li> Export all ORDS modules from the current schema</li> Export all object DDL from the current schema</li> Export table data into CSV files</li> Provide basic script templates for export/import of whole app for DEV, TEST and PROD</li> Everything in a (hopefully) nice directory structure ready to use with version control</li> Return value is a file collection of type plex.tab_export_files (it was apex_t_export_files before PLEX version 2) for further processing Each file in the collection is represented by a record with two columns name</code> of type VARCHAR2(255), which is in fact the file path</li> contents</code> of type CLOB</li> </ul> </li> You can optionally zip the file collection with the helper function to_zip</code></li> Also see the my previous post on how to handle the apex_t_export_files type returned by the APEX_EXPORT package with SQL*Plus</a></li> </ul> </li> </ul> Getting Started</h2> Download the latest code</a></li> Run the provided install script plex_install.sql</code> (provides compiler flags) in your desired schema - could also be a central tools schema, don’t forget grant execute on plex to xxx</code></li> Startup your favorite SQL Tool, connect to your app schema and fire up the following query:</li> </ol> select plex.to_zip(plex.backapp(p_app_id => yourAppId)) from dual;</code></pre> Save the resulting BLOB file under a name with the extension .zip</code> and extract it to a local directory of your choice. You will find this directory structure and files: - app_backend (only, when p_include_object_ddl is set to true, see next example) - package_bodies - packages - tables - ref_constraints - ... - app_data (only, when p_include_data is set to true) - app_frontend (for the apex app files without subfolder fxxx) - pages - shared_components - ... - app_web_services (only, when p_include_ords_modules is set to true) - docs - scripts - logs - templates - 1_export_app_from_DEV.bat - 2_install_app_into_TEST.bat - 3_install_app_into_PROD.bat - export_app_custom_code.sql - install_app_custom_code.sql - install_backend_generated_by_plex.sql - install_frontend_generated_by_apex.sql - install_web_services_generated_by_ords.sql - tests - plex_README.md - plex_runtime_log.md</code></pre> If you like, you can fully configure your first export into the zip file. The plex.backapp</code> method has boolean parameters, so you need to use an inline function in a pure SQL context. You can also use an anonymous PL/SQL block or you create a small SQL wrapper for the method like the inline function of the example. All parameters are optional and listed here with their default values: -- Inline function because of boolean parameters (needs Oracle 12c or higher). -- Alternative create a helper function and call that in a SQL context. WITH FUNCTION backapp RETURN BLOB IS BEGIN RETURN plex.to_zip(plex.backapp( -- All parameters are optional and shown with their defaults -- APEX App (only available, when APEX is installed): p_app_id => null, -- If null, we simply skip the APEX app export. p_app_date => true, -- If true, include export date and time in the result. p_app_public_reports => true, -- If true, include public reports that a user saved. p_app_private_reports => false, -- If true, include private reports that a user saved. p_app_notifications => false, -- If true, include report notifications. p_app_translations => true, -- If true, include application translation mappings and all text from the translation repository. p_app_pkg_app_mapping => false, -- If true, export installed packaged applications with references to the packaged application definition. If FALSE, export them as normal applications. p_app_original_ids => false, -- If true, export with the IDs as they were when the application was imported. p_app_subscriptions => true, -- If true, components contain subscription references. p_app_comments => true, -- If true, include developer comments. p_app_supporting_objects => null, -- If 'Y', export supporting objects. If 'I', automatically install on import. If 'N', do not export supporting objects. If null, the application's include in export deployment value is used. p_app_include_single_file => false, -- If true, the single sql install file is also included beside the splitted files. p_app_build_status_run_only => false, -- If true, the build status of the app will be overwritten to RUN_ONLY. -- ORDS Modules (only available, when ORDS is installed): p_include_ords_modules => false, -- If true, include ORDS modules of current user/schema. -- Schema Objects: p_include_object_ddl => false, -- If true, include DDL of current user/schema and all its objects. p_object_type_like => null, -- A comma separated list of like expressions to filter the objects - example: '%BODY,JAVA%' will be translated to: ... from user_objects where ... and (object_type like '%BODY' escape '\' or object_type like 'JAVA%' escape '\'). p_object_type_not_like => null, -- A comma separated list of not like expressions to filter the objects - example: '%BODY,JAVA%' will be translated to: ... from user_objects where ... and (object_type not like '%BODY' escape '\' and object_type not like 'JAVA%' escape '\'). p_object_name_like => null, -- A comma separated list of like expressions to filter the objects - example: 'EMP%,DEPT%' will be translated to: ... from user_objects where ... and (object_name like 'EMP%' escape '\' or object_name like 'DEPT%' escape '\'). p_object_name_not_like => null, -- A comma separated list of not like expressions to filter the objects - example: 'EMP%,DEPT%' will be translated to: ... from user_objects where ... and (object_name not like 'EMP%' escape '\' and object_name not like 'DEPT%' escape '\'). p_object_view_remove_col_list => true, -- If true, the outer column list, added by Oracle on views during compilation, is removed -- Table Data: p_include_data => false, -- If true, include CSV data of each table. p_data_as_of_minutes_ago => 0, -- Read consistent data with the resulting timestamp(SCN). p_data_max_rows => 1000, -- Maximum number of rows per table. p_data_table_name_like => null, -- A comma separated list of like expressions to filter the tables - example: 'EMP%,DEPT%' will be translated to: where ... and (table_name like 'EMP%' escape '\' or table_name like 'DEPT%' escape '\'). p_data_table_name_not_like => null, -- A comma separated list of not like expressions to filter the tables - example: 'EMP%,DEPT%' will be translated to: where ... and (table_name not like 'EMP%' escape '\' and table_name not like 'DEPT%' escape '\'). -- General Options: p_include_templates => true, -- If true, include templates for README.md, export and install scripts. p_include_runtime_log => true, -- If true, generate file plex_runtime_log.md with detailed runtime infos. p_include_error_log => true, -- If true, generate file plex_error_log.md with detailed error messages. p_base_path_backend => 'app_backend', -- The base path in the project root for the Schema objects. p_base_path_frontend => 'app_frontend', -- The base path in the project root for the APEX app. p_base_path_web_services => 'app_web_services', -- The base path in the project root for the ORDS modules. p_base_path_data => 'app_data')); -- The base path in the project root for the table data. END backapp; SELECT backapp FROM dual;</code></pre> ATTENTION: Exporting all database objects can take some time. I have seen huge runtime differences from 6 seconds for a small app up to several hundred seconds for big apps and/or slow databases. This is normally not the problem of PLEX. If you are interested in runtime statistics of PLEX, you can inspect the delivered plex_runtime_log.md</code> in the root directory. Also, the possibility to export the data of your tables into CSV files does not mean that you should do this without thinking about it. The main reason for me to implement this feature was to track changes on catalog tables by regularly calling this export feature with a sensitive table filter and max rows parameter as catalog data is often relevant in business logic. If you have organized your app into multiple schemas as described in The Pink Database Paradigm</a>, you may need to export database objects from more then one schema. This is no problem for plex.backapp</code> as all parameters are optional - you can simply logon to your second or third schema and extract only the DDL for these schemas by omitting the p_app_id</code> parameter and setting p_include_object_ddl</code> to true</code>. Then unload the DDL files into a different directory - for example app_backend_schemaName</code>. A last word: you should inspect all the exported files and scripts and check if this solution can work for you. If not, please let me know what is missing or what should be done in a different way … Feedback is welcome - simply create a new issue</a> at the GitHub project page</a> Next Steps</h2> It is up to you how you organize the version control repository and how often you export your APEX app or object DDL. I would follow the files first approach and extract the object DDL only ones to have a starting point. The APEX application needs regular exports - if you like, you can automate this. Following the files first approach is sometimes not easy when you are using low code tools like Quick SQL</a> and Blueprint</a> in APEX or code generators like OraMUC’s Table API Generator</a>. There could be a need to regularly extract (maybe unknown) objects (not created by yourself) into version control to understand and document what you got from others (people or generators)… If the directory structure provided by PLEX does not match your needs - no problem - you can align it. Simply loop over the returned file collection and do your necessary work - here an example: DECLARE l_files plex.tab_export_files; -- before PLEX v2 it was apex_t_export_files BEGIN l_files := plex.backapp(p_app_id => 100); FOR i IN 1..l_files.count LOOP -- relocate APEX app files from app_frontend to app_ui IF l_files(i).name LIKE 'app_frontend/%' THEN l_files(i).name := replace(l_files(i).name, 'app_frontend/', 'app_ui/'); l_files(i).contents := replace(l_files(i).contents, 'prompt --app_frontend/', 'prompt --app_ui/'); END IF; -- correct file links in install script IF l_files(i).name = 'scripts/install_frontend_generated_by_apex.sql' THEN l_files(i).contents := replace(l_files(i).contents, '@../app_frontend/', '@../app_ui/'); END IF; END LOOP; -- more alignments... END;</code></pre> For unloading the resulting file collection with SQL*Plus, please have a look in the scripts/templates</code> folder of your export - there are examples to do this. See also my previous post on how to handle the apex_t_export_files type returned by the APEX_EXPORT package with SQL*Plus</a>. Some people prefer to devide their DDL scripts into the two categories restartable (like packages) and run once (like tables). Others like to have their scripts in a way that they are always restartable and the DDL script itself takes care about doing the work only once when needed. The advantage of the second way is that your backend install/deployment script is always the same and it simply calls all objects DDL scripts. There is no right or wrong in doing it this or that way - each project/team has its specific requirements and history. The important thing is, that you start to use a version control system to be able to log your changes and document your code. By the way - PLEX provides script templates and object DDL that follows the second approach: You can always have the same install/deployment script and the DDL scripts are restartable - check it out by looking in one of your exported table DDL scripts. You are now at the point where PLEX can’t do anything more for you. If you like to export your object DDL scripts more often, you have to find a way to be able to protect some of your scripts against overwriting. Imagine you had to add two columns to a table and you provided a restartable alter statement for this in the existing DDL script. If you export this table script the next time with PLEX (or with dbms_metadata.get_ddl, which is used in the background), your alter statements are gone and the new columns are simply listed in the create table statement. With this script you are not be able to deploy your changes to TEST or PROD. One solution is to copy the original table script and name it e.g. EMPLOYEES.dev.sql</code>. In this script you maintain the restartable alter statements. If you run plex.backapp</code> again you are overwrite save. The script EMPLOYEES.sql</code> reflects your current table definition and can still be executed - it does nothing because the table is already existing. The script EMPLOYEES.dev.sql</code> reflects your development history and need to be added to your custom install/deployment script. As you can see, PLEX can do only the basics for you. It is up to the developers how they manage their version control repository and how they do their deployments - there are thousends of ways to do it … Inspirations / Further Reading</h2> Thanks are going to: André Borngräber for his ability to think and discuss database topics in deep details</li> Blain Carter for his thoughts on CI/CD for Database Developers – Export Database Objects into Version Control</a></li> Markus Dötsch for the first BackApp tests and cross-reading the first version of this post</li> Martin D’Souza for his time and the interesting discussion about code generation and version control at the APEX Connect 2018 in Düsseldorf, Germany and his blog post Exporting APEX Application in SQLcl with Build Status Override</a> - plex.backapp</code> has now a parameter for this ;-)</li> Philipp Salvisberg for his thoughts on The Pink Database Paradigm</a></li> Samuel Nitsche for his thoughts on There is no clean (database) development without Version Control</a></li> Tim Hall for his article about Generating CSV Files</a></li> </ul> Thats It</h2> Hopefully PLEX is usefull for someone else. Happy coding, apexing, version controlling :-) Ottmar APEX_EXPORT and Version Control 2018-07-25T00:00:00+00:00 Since years it has been possible to export an APEX app definition with the help of APEXExport, a Java utility delivered within the APEX install zip file. There is also the possibility to split the file into its components like pages, plugins and so on. There are some blog postings available how to do this - simply ask Google</a>. Also the Java based SQLcl has the capability to do the export of an APEX app directly. So why bother with a different way to export and split an APEX application? Since APEX 5.1.4 there is a new PL/SQL package APEX_EXPORT, which can be used to get a file collection of the application - one big file or the splitted ones. Unfortunately in the API docs</a> there is (as of this writing) only one example available how to export the single file within SQL*Plus - no example to handle the splitted files. But again, why discuss this if there are already options to do it? Because of the possibility to modifiy the APEX_EXPORT file collection before fetching it into the file system. Imagine you have a different repository structure and the delivered file structure of the splitted files does not match your needs or you want to have all install files in one scripts directory of your repo and therefore need a relocation of the app install script. Another use case is to enrich the file collection with additional data or objects. This was possible in the past also with some postprocessing outside the database, but now we are able to do this within our DB session and PL/SQL. I have already started an open source project to leaverage these possibilities - more about this in my next post… So how to do it? First, I have to describe what the sctructure of each file in the collection is. Dead simple: a record type with two columns: name</code> of type VARCHAR2(255) which is in fact the file path and contents</code> of type CLOB. The desired file structure: - app_backend - app_frontend (for the splitted files without subfolder fxxx) - pages - shared_components - ... - docs - scripts - logs - temp_export_files.sql (our intermediate script file) - export_frontend_from_DEV_20180722_2045.log (one of our export logs) - ... - export_frontend.bat (our OS shell script to start the export) - export_frontend.sql (our export script) - install_frontend.sql (the generated install file from apex_export) - tests - README.md</code></pre> Here comes the idea: We create a script file to get the file collection, iterate over the collection and modify the content regarding our needs</li> To unload the files with the spool command in SQL*Plus we need it accessible via SQL - therefore we put the files into a global temporary table</li> We need to create an intermediate script file to unload the files (select the clob content)</li> We also need to create host commands for the needed directories because the spool command does NOT create missing directories</li> We spool our progress to a log file for later reference</li> </ul> -- file: export_frontend.sql set verify off feedback off heading off set trimout on trimspool on pagesize 0 linesize 5000 long 100000000 longchunksize 32767 whenever sqlerror exit sql.sqlcode rollback -- https://blogs.oracle.com/opal/sqlplus-101-substitution-variables define logfile = "logs/export_frontend_from_&2._&3._&4..log" spool "&logfile." replace prompt prompt Start frontend export for app &1. on &2. prompt ================================================== prompt Create global temporary table temp_export_files if not exist BEGIN FOR i IN (SELECT 'TEMP_EXPORT_FILES' AS object_name FROM dual MINUS SELECT object_name FROM user_objects) LOOP EXECUTE IMMEDIATE q'[ -------------------------------------------------------------------------------- CREATE GLOBAL TEMPORARY TABLE temp_export_files ( name VARCHAR2(255), contents CLOB ) ON COMMIT DELETE ROWS -------------------------------------------------------------------------------- ]'; END LOOP; END; / prompt Do the frontend export, relocate files and save to temporary table DECLARE l_app_id pls_integer := &1.; l_files apex_t_export_files; BEGIN l_files := apex_export.get_application( p_application_id => l_app_id, p_split => true, p_with_date => true, p_with_ir_public_reports => true, p_with_ir_private_reports => false, p_with_ir_notifications => false, p_with_translations => true, p_with_pkg_app_mapping => false, p_with_original_ids => true, p_with_no_subscriptions => false, p_with_comments => true, p_with_supporting_objects => 'Y'); FOR i IN 1..l_files.count LOOP -- relocate files to own project structure l_files(i).name := replace( l_files(i).name, 'f'||l_app_id||'/application/', '../app_frontend/' ); -- correct prompts for relocation l_files(i).contents := replace( l_files(i).contents, 'prompt --application/', 'prompt --app_frontend/' ); -- special handling for install file IF l_files(i).name = 'f'||l_app_id||'/install.sql' THEN l_files(i).name := 'install_frontend.sql'; l_files(i).contents := replace( replace( l_files(i).contents, '@application/', '@../app_frontend/' ), 'prompt --install', 'prompt --install_frontend' ); END IF; END LOOP; FORALL i IN 1..l_files.count INSERT INTO temp_export_files VALUES ( l_files(i).name, l_files(i).contents ); END; / prompt Create intermediate script file to unload the table content into files spool off set termout off serveroutput on spool "logs/temp_export_files.sql" BEGIN -- create host commands for the needed directories (spool does NOT create missing directories) FOR i IN ( WITH t1 AS ( SELECT regexp_substr(name, '^((\w|\.)+\/)+') AS dir FROM temp_export_files ) SELECT DISTINCT dir, -- This is for Windows to create a directory and suppress warning if it exist. -- Align the command to your operating system: 'host mkdir "' || replace(dir,'/','\') || '" 2>NUL' AS mkdir FROM t1 WHERE dir IS NOT NULL ) LOOP dbms_output.put_line('set termout on'); dbms_output.put_line('spool "&logfile." append'); dbms_output.put_line('prompt --create directory if not exist: ' || i.dir); dbms_output.put_line('spool off'); dbms_output.put_line('set termout off'); dbms_output.put_line(i.mkdir); dbms_output.put_line('-----'); END LOOP; -- create the spool calls for unload the files FOR i IN (SELECT * FROM temp_export_files) LOOP dbms_output.put_line('set termout on'); dbms_output.put_line('spool "&logfile." append'); dbms_output.put_line('prompt --' || i.name); dbms_output.put_line('spool off'); dbms_output.put_line('set termout off'); dbms_output.put_line('spool "' || i.name || '"'); dbms_output.put_line('select contents from temp_export_files where name = ''' || i.name || ''';'); dbms_output.put_line('spool off'); dbms_output.put_line('-----'); END LOOP; END; / spool off set termout on serveroutput off spool "&logfile." append prompt Call the intermediate script file to save the files spool off @logs/temp_export_files.sql set termout on serveroutput off spool "&logfile." append prompt Delete files from the global temporary table COMMIT; prompt ================================================== prompt Export DONE :-) prompt</code></pre> To run this script file on your operating system, you need a shell script to call it. Here is an example for Windows: rem file: export_frontend.bat echo off setlocal set systemrole=DEV set connection=localhost:1521/orcl set schema=HR set app_id=100 set areyousure=N rem align delimiters to your os locale for /f "tokens=1-3 delims=. " %%a in ('date /t') do (set mydate=%%c%%b%%a) for /f "tokens=1-2 delims=:" %%a in ('time /t') do (set mytime=%%a%%b) :PROMPT echo. echo. set /p areyousure=Export %schema% app %app_id% from %systemrole% (Y/N)? if /i %areyousure% neq y goto END set NLS_LANG=AMERICAN_AMERICA.AL32UTF8 set /p password=Please enter password for %schema% on %systemrole%: echo exit | sqlplus -S %schema%/%password%@%connection% ^ @export_frontend.sql ^ %app_id% ^ %systemrole% ^ %mydate% ^ %mytime% :END pause</code></pre> There are better ways then to ask for a password at runtime - but that is not the focus of this post. Also, you may not want to keep the command line open until the user presses a key in a fully automated setup. This is only to be able to see any errors before the shell window closes when opened via double click on the file. Hope this helps someone. Happy exporting :-) Ottmar Show Today Button in APEX Datepicker 2018-07-22T00:00:00+00:00 Maxime Tremblay has already described here</a> how to customize the datepicker. He had to deal with missing classes after changing options. I found this week a different way to customize the datepicker without loosing classes added by APEX: Simply by setting datepicker defaults on page load before any datepicker is initialized. In my case I wanted to enable the button panel which shows two buttons - one to go to the current month which is called Today</code> and a second one to close the datepicker which is called Done</code> I only wanted the today button, so I had to provide beside the JavaScript or dynamic action some CSS to hide the second one: apex.jQuery(document).ready(function(){ $.datepicker.setDefaults({ 'showButtonPanel': true }); });</code></pre>body .ui-datepicker-close { display: none !important; }</code></pre> Just for reference: The jQuery UI datepicker widget can be tested here</a>. Be careful to pick the correct version of the jQuery UI API docs for your specific APEX version. Happy coding :-) Ottmar HydeBar 2017-11-29T00:00:00+00:00 Earlier this year I wrote at the end of this post</a>: PS: In the meantime I switched complete to Markdown and Jekyll</a>, a static site generator - but this is another story for another post … </blockquote> As always - it took a bit longer then expected. In the meantime I was fiddling around with Jekyll and modifying the default theme Minima to my needs. Then I stumbled over the JSDoc theme Minami</a> and the Liquid docs site</a> - both with nice sidebars. I wanted to have such a nice sidebar for my dev blog and began to look deeper how it was implemented. To my surprise the sidebar from the JSDoc theme Minami was complete CSS driven - cool. It took me a while to full understand how it was working. With this knowledge and the design inspirations I was able to improve the Jekyll default theme to a sidebar template. As a bonus I integrated the HTML based slideshow library Reveal.js - you can write now your blog posts and slides with markdown. The nice thing about blogging with markdown is, that you don’t need to switch your tools - you use the usual ones from the daily programming job. It goes even better when you use GitHub: You commit your changes and GitHub takes care about generating your blog content from your sources. And the result is a fast and secure static site. I had a accepted talk at the DOAG Conference + Exhibition 2017</a> in the stream “Strategy & Business Practices” about blogging for developers with Jekyll. Unfortunately short before I had an accident and broke my right upper arm :-( My colleague Markus Dötsch was taking over the presentation and I concentrated on delivering the free time project “Jekyll sidebar template”. This was the working title before I renamed it to “HydeBar” - a play with “Dr. Jekyll and Mr. Hyde” and the fact, that it is a sidebar template. Yesterday I released version 1.0.0. You can start blogging for free in five minutes with this quick start guide</a>. The online demo</a> is serving as template, documentation and showcase. The mentioned slides from the DOAG Conference</a> are included in this demo (in German, sorry…). Happy blogging :-) Ottmar Form Behaviour - Bug or Feature? 2017-09-26T00:00:00+00:00 Today I stumbled over this strange behaviour of an APEX page: The page was always submitted when I pressed the enter key in a search field to force an AJAX refresh on a report. The item property to submit the page on pressing enter was switched off. I asked my collegue Markus, who had this error also some time ago (I could not remember the solution, although he told me about this issue). The reason was that the search field was the only input item on that page. After a short Google search I found the “root cause” for this “bug”, or should I say “feature”? This behaviour is described in the HTML 2.0 specification (section 8.2)</a>. To workaround this you can simply create a second input item and hide it by setting the “HTML Form Element Attributes” to style="display:none;"</code>. This was also posted by Denes Kubicek</a> back in 2008. I like the idea of one of the Stack Overflow comments</a> to put anywhere into the form (in APEX in fact the whole page, e.g. the header or footer of a region) this HTML snippet and not to create an “useless” page item: <input type="text" name="StackOverflow1370021" style="display:none;" /></code></pre> It is worth to mention that I run the newest Chrome browser and this “feature” is still available… Happy coding :-) Ottmar Oracle DB 11gXE Install File Swap Check Disabler 2017-06-13T00:00:00+00:00 Many people have problems to install Oracle 11XE in a Docker environment because the install file checks the available swap space in the container. In a container environment this fails often - see here</a> or here</a>, because the swap space is optimized for the entire stack and not controlled from within the operating system of the container. We have to disable the swap space check in the installation file. I wrote another blog post about this</a>. The problem is here, that you need a Linux based system to do the necessary steps. Under Windows you have no chance and you have to do it by yourself because for license reasons everyone has to download his/her own copy of the install file from Oracle OTN</a>. I came up with the idea to simply do all the steps in a Docker container under the same Linux (oraclelinux:7-slim) which is later on needed with the official Oracle Docker file</a> for an XE instance. With this solution you are able to prepare the install file more or less automatically under every operating system, which can run Docker - also under Windows. For more Details see the project on GitHub</a>. Happy installing :-) Ottmar jQuery Plugin svg2img 2017-04-03T00:00:00+00:00 SVG based charts and visualizations are often used these days. But what if you want to use your browser inline SVGs, generated with some sort of JavaScript and fancy styled with CSS, offline - maybe in a presentation, send by email or printed out large scaled? You can create a screenshot, of course. But does this look nice when it comes to scaling? You can try to copy the HTML code of the SVG and wrap it into a correct SVG container, but then you will loose in the most cases the stylings, as normally not all CSS definitions are directly attached to the SVG elements. You can also use a browser extension like this one for Google Chrome</a>, but then you depend on the browser and the extension. There is now a declarative way with the help of a jQuery plugin, which is 100% client based: jQuery('#example-graph').svg2img();</code></pre> There is one default option - you can redeclare it: jQuery.fn.svg2img.defaults = { debug: false // write debug information to console };</code></pre> You can also set the debug option at runtime: $('#example-graph').svg2img({debug:true});</code></pre> The plugin checks if your selector is a SVG element. If this is not the case, it searches under each selector element for all SVGs and exports it. The name(s) of the file(s) are automatically derived from the element ID (or parent element ID) or set to export</code>. The current date and time is appended to the file name, for example example-graph-20170403-154653.svg</code>. You can download all SVGs from one page by setting the selector to the document, body or svg - as you like it. There is no need for a each()</code> call, svg2img</code> works on the whole selection and is chainable: $("svg").svg2img().css("border", "1px solid red");</code></pre> If you want to use anchors to provide export links then it is recommended to prevent the default action like so: <a href="" onclick="event.preventDefault(); $('#example-graph').svg2img();">SVG</a></code></pre> Otherwise Firefox and IE failing to save. One last thing: Safari has currently (as of this writing) problems with the underlaying savAs()</code> implementation and tries to open the images in a new tab with or without success. See also this issue on GitHub</a>. UPDATE 2017-04-14: After update to macOS Sierra 10.12.4 Safari 10.1 works also - but is as slow as before - collecting CSS styles runs on my MacBook 2800ms in Safari, 20ms in Google Chrome :-( I stopped to support other image formats then SVG since the canvas export behind the scenes was not really working in too many browsers and the core feature was and is to convert inline SVGs to standalone SVG images. The project is hosted on GitHub</a> and MIT licensed. Happy SVG exporting :-) Ottmar Pitfalls with Oracle 11g XE and Docker on Mac OS 2017-03-21T00:00:00+00:00 Recently I tried to install Oracle XE into a docker container on Mac OS Sierra using the official docker file</a> from Oracle. I had no luck, because the installation failed with this error message: This system does not meet the minimum requirements for swap space. Based on the amount of physical memory available on the system, Oracle Database 11g Express Edition requires 2048 MB of swap space. This system has 0 MB of swap space. Configure more swap space on the system and retry the installation. </blockquote> The problem is, that you can’t do anything for this on a Mac - the system manages the swap files by itself and if you need for an example 64 GB you will get it. My new standard MacBook has 8 GB RAM and there is currently no reason for a swap file. So, what can we do? After a short search I found this blog post</a>, which describes how to alter the pre-installation script of the RPM installation file with the help of the repmrebuild</code> command. Unfortunately this command is not available on Mac OS. I found only rpmbuild as part of the rpm installation with Homebrew</a>. Luckily I found another way to modify a rpm package under Mac OS: There is a Ruby based tool to build packages for multiple platforms called fpm</a>. So, lets start: If you don’t have Homebrew</a> on your Mac, install this first: /usr/bin/ruby -e \ "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"</code></pre> Install rpm: brew install rpm</code></pre> Install tar: brew install gnu-tar</code></pre> Install fpm: gem install --no-ri --no-rdoc fpm</code></pre> Unzip the downloaded Linux XE installation archive and go into the directory Disk1: unzip oracle-xe-11.2.0-1.0.x86_64.rpm.zip && \ cd Disk1</code></pre> Write rpm scripts to text file: rpm \ --scripts \ -qp oracle-xe-11.2.0-1.0.x86_64.rpm \ > scripts.txt</code></pre> Find this loop (from line 197 until line 212 as of this writing) and comment out or delete it: # check and disallow install, if swap space is less than Min( 2047, 2 * RAM)` if [ $swapspace -lt $requiredswapspace ] then if [ "$requiredswapspace" = "2047" ]; then requiredswapspace=2048 fi echo echo "This system does not meet the minimum requirements for swap space. Based on the amount of physical memory available on the system, Oracle Database 11g Express Edition requires $requiredswapspace MB of swap space. This system has $swapspace MB of swap space. Configure more swap space on the system and retry the installation." echo exit 1 fi</code></pre> Find this line (starting on line 328 as of this writing) and delete the line and everything behind until the file end: postinstall scriptlet (using /bin/sh):</code></pre> Delete the very first line containing this code: preinstall scriptlet (using /bin/sh):</code></pre> Save the file and bring it back into the rpm with this command (will take some time): fpm \ -s rpm \ -t rpm \ -f --before-install scripts.txt \ ./oracle-xe-11.2.0-1.0.x86_64.rpm</code></pre> Go out of Disk1 and update the zip archive, because the Oracle provided build script needs it compressed (will take some time): cd .. && \ zip -r oracle-xe-11.2.0-1.0.x86_64.rpm.zip Disk1</code></pre> When you now do the Docker build you will face another error message: package oracle-xe-11.2.0-1.0.x86_64 is intended for a different operating system </blockquote> To fix this is fairly easy. Open the Dockerfile.xe, go to line 61 (as of this writing) and change this code: rpm -i Disk1/*.rpm && \</code></pre> to this one: rpm -i --ignoreos Disk1/*.rpm && \</code></pre> When you now start the docker build you should be successful (will take some time): sudo ./buildDockerImage.sh -v 11.2.0.2 -x -i</code></pre> The last step is to run the container - on the first startup the database is created (will take some time, align the name to your needs): docker run \ --name orclapex51 \ --shm-size=1g \ -p 1521:1521 \ -p 8080:8080 \ oracle/database:11.2.0.2-xe</code></pre> If you plan to use APEX without any dedicated webserver (maybe because you are the only user and it is only a small dev instance) you should pimp the EPG a little bit: alter system set shared_servers=10 scope=both; alter system set max_shared_servers=20 scope=both; alter system set dispatchers='(PROTOCOL=TCP) (SERVICE=xeXDB) (DISPATCHERS=2)' scope=both;</code></pre> I use currently a Node.js based proxy server running directly on my Mac OS, because I need a webserver there anyway. The nice thing is, that the EPG don’t need to serve the images and I also don’t need to install the images into the EPG, when I upgrade to a newer APEX version. This saves me time and the EPG runs faster because of less number of requests. Happy installing :-) Ottmar APEX Plugin dhtmlxGantt 2017-03-14T00:00:00+00:00 Some weeks ago I was asked, if I can create a plugin for the Gantt chart library from dhtmlx.com</a>. I had a look at the library and I thought that it should be possible. Because all my plugin developments are free time projects it took a while. Now the first public version is available. It is based on the GPLv2 version of the library, which has a reduced set of functionality. If you need advanced features like milestones, auto scheduling or calculating the critical path you can buy a Pro version of the library. I have no relation to the company behind dhtmlx.com, so please do not complain. As I told before I was asked for this plugin (which will be used in a Pro version there…) and I like the visualization from DHTMLX. You can find the sources and more informations on GitHub</a> and a online demo on apex.oracle.com</a>. Maybe the plugin is helpful for someone else… Happy project planning :-) Ottmar DokuWiki Plugin Reveal.js 2017-03-05T00:00:00+00:00 DokuWiki</a> is very good for creating documentation in a team. It is PHP based, needs less resources and is running without a database only on files on nearly every webserver. If your infrastructure is crashed for whatever reason and you have at least access to the files (on the webserver or in a backup) you can read your documentation with a simple text editor, because DokuWiki’s syntax is designed to be readable as possible without any rendering to HTML. For the APEX connect 2015 I started to use Markdown and Pandoc</a> to create HTML based slides. Pandoc supports different HTML slideshow tools - two of them are also available as plugins in DokuWiki: S5 and Reveal.js</a>. Reveal.js is the more modern library and I used this one together with Pandoc. For the DOAG conference last year I tried to use Reveals.js as a DokuWiki Plugin - I wanted to have at the same time a normal wiki page and the possibility to render this page as a slideshow. Unfortunately the existing plugin was only a base implementation and many features from Reveal.js were not available. I decided to contribute to the existing plugin. The list of my improvements is long - I had fun and learned a lot. The plugin can be found here on the DokuWiki homepage</a>. Happy presenting :-) Ottmar PS: In the meantime I switched complete to Markdown and Jekyll</a>, a static site generator - but this is another story for another post … Markdown Reporter 2017-01-23T00:00:00+00:00 On the APEX connect 2016 in Berlin I talked about my APEX Markdown plugin and fine art printing with the help of Pandoc and LaTeX. When I was preparing my slides and documenting a SQL query in a Markdown code block I asked myself: What, if the database is taking this Markdown code block and runs the query inside to convert it into data? And what, if I take this Markdown text with the data in the code blocks and post it to a small webservice which converts the data blocks into charts and the whole document into HTML or a LaTeX based PDF? The idea to Markdown Reporter was born. I talked about this project on the DOAG conference in Nuremberg last November. Here are the slides</a> (in german, sorry). If you want to know how it looks like - here is a PDF demo report</a> The implementation has three main components: An Oracle PL/SQL Package for preprocessing your data and communicating with the printserver</li> A Node.js based printserver, which is in fact a web based remote shell for the format converter Pandoc</a></li> A Pandoc filter written in Python, which intercepts the converting process and generates the charts based on the preprocessed CSV data with the help of Pythons matplotlib</a></li> </ul> The HTML and PDF output is complete vector based and all used tools are open source. You can find the sources and more informations on GitHub</a>. Happy reporting :-) Ottmar Table API Generator 2016-06-10T00:00:00+00:00 Last year I needed for a project a simple table API, which is able to generate a generic change log - something like “show me for table xyz which column was changed when and what was the old value and what is the new one”. Only changed values should be stored and all tables should log into the same generic logging table. Sounds like we need a generator for this. Unfortunately there was no time in the project to create such a generator from the ground. Luckily André, a colleague of me, had started to implement such an API generator in his free time and I had only to implement the generic change log functionality - also in my free time. Our API generator open source project was born :-) Later on I needed a function to get the id from an record by the unique columns - next feature for the API generator: generate a (overloaded) function for each unique constraint on a table. Now it is a simple function call to get the id: your_var := your_table_name_api.get_pk_by_unique_cols( p_column_name_1 => 'xyz', p_column_name_2 => 123, ...);</code></pre> The API is also handling the case, when no data is found and everything without the need to look in the dictionary for the unique columns of that table or to write boilerplate code and possibly forget one column. The next thing for APEX was “For tabular forms you can’t use APIs”. We came up with the idea to generate a view for each table with an instead of trigger for inserts, updates and deletions. The trigger is simply calling the API. Now you can use APEX own “automatic row processing” and the generic logging together. In my holiday this spring I stumbled over a tweet from Phillip Salvisberg about his oddgen project</a>, a SQL Developer extension to invoke dictionary-driven code generators. I was thrilled, we can have a SQL Developer integration by writing a small PL/SQL wrapper for oddgen. I finished the integration in my holiday, I could not wait. Over the time we found out, that the generation of the thin API wrappers for our tables saves us time and makes our code more stable and readable because of the saved boilerplate code. Business logic packages becomes invalid after table changes and API regeneration. This is good, because you can see, which API calls are now invalid. If you have simple insert or update statements you might oversee some places, where the statement is still valid for example with an added table column, but in fact does the wrong insert or update. And the last thing, what is now possible: Security - separate the data from the user interface. Simply grant execute rights on the API and business logic packages to the UI schema and disable the deletion of rows in your API packages when you don’t need it. No user will be able to drop or truncate your tables, because there are no tables in the UI schema. The generated API packages are really thin - nothing special at all. Nevertheless they are powerful time savers. You can find the sources and more informations on GitHub</a>. Happy coding :-) Ottmar Markdown for Oracle APEX 2016-01-01T00:00:00+00:00 Some years ago I implemented a task board in APEX. For commenting I used the stackexchange</a> markdown implementation</a> (old Google code repo</a>), which is a pure JavaScript converter and editor and based on showdown.js</a>. Some months ago I had to implement an application, which should be able to have multiple editors on one page and the editors should be able to support a read only mode. Furthermore there were hard limits for the amount of text for each editor. Both things which are not so easy to implement with the rich text editor in APEX - no really read only mode and the amount of text is also a problem because of the HTML overhead. It was time to rethink my old implementation and to build a plugin, which creates for each text area with the class markdown an own editor. All other non input elements with a class markdown should simply rendered to HTML, which is then also the solution for the read only mode of the editors. APEX standard text area items supports a read only mode and the class markdown from the text area is also rendered by the APEX engine for this mode :-) The nice side effect is, that I don’t had to reinvent the wheel by creating an own item type - I could reuse the standard APEX text area item and things like the character counter working out of the box. You can find the sources and more informations on GitHub</a> and a demo app on apex.oracle.com</a>. Happy coding :-) Ottmar Why a subdomain for an APEX development system is a bad idea 2015-12-12T00:00:00+00:00 Do you find dev.apex.mycompany.tld a nice, rememberable address? Unfortunately this will not work - at least when you use Internet Explorer (which is the standard browser in most companies) and you use the same cookie name in some or all applications to share the session across multiple applications. What is the problem on a subdomain?</h2> Internet Explorer including version 11 has a really bad cookie implementation. If you login to your productive APEX instance under apex.mycompany.tld and then to your development instance under dev.apex.mycompany.tld, IE sends the session cookie from your productive instance to your development instance. You will not be able to login - the server is generating a new session each time because of the invalid session cookie. It comes to a nightmare if you configured your systems for automatic login with the windows login credentials as described in this document</a> from Niels de Bruijn. The automatic login leads to hundreds new sessions in a few seconds - depending on the speed of your development system. Not to mention that Chrome and Firefox are working well… More details about IE’s cookie desaster can be found on blogs.msdn.com</a> - the relevant question for this blog entry is number three. Happy APEXing :-) Ottmar D3.js Force-Directed Network Chart 2015-02-20T00:00:00+00:00 My first Oracle APEX plugin and open source project :-) Everything started some months ago when I stumbled over this fascinating force-directed graph</a>. I immediately had the idea to use this technology to visualize data models in APEX. My experience is, that direct after I documented a data model, this documentation is out of date. I wanted to have a live visualization of the data models. After the first implementation it turned out, that the resulting graph is also useful for other purposes and I decided to build up an APEX plugin for easier integration. If you use the bare JavaScript files you can run it on every HTML page: <div id="example"></div> <link href="/d3.js/d3-force-3.0.0.css" rel="stylesheet" type="text/css"> <script src="/d3.js/ResizeObserver-1.5.0.min.js"></script> <script src="/d3.js/d3-3.5.6.min.js"></script> <script src="/d3.js/d3-force-3.0.0.min.js"></script> <script> window.onload = function() { window.example = netGobrechtsD3Force('example') .height(400) .width(800) .useDomParentWidth(true) //for responsive layout //.zoomMode(true) .lassoMode(true) .wrapLabels(true) .showBorder(false) .debug(true) //to enable the customization wizard .render(); //sample data is provided when called without data //also see https://ogobrecht.github.io/d3-force-apex-plugin/tutorial-1-getting-started.html } </script></code></pre> I found myself often fiddling around with the parameters of the physical simulation - change config, reload page, not amused, change config… Really annoying - even I know the implementation behind. So, the question was: What can I do for an easy configuration. My answer was after a while of thinking and trying out: A customization wizard with a live preview of the changes. The link to enter the wizard is shown when the debug mode is switched on or in APEX when the developer bar is shown - try it out on the getting started page of the API docs</a>. You can find the sources and more informations on GitHub</a> and a demo app on apex.oracle.com</a>. Happy networking :-) Ottmar Double click on APEX report row opens first link target 2015-01-14T00:00:00+00:00 Do you like scanning a report for certain information and then moving the mouse to the first column to click the small link for details or a edit form? Would it be nice to double click anywhere on the row to open the first link? With a little help of a dynamic action on page zero you have the solution for all your interactive and standard reports in your application. The only additional to do is, to find out the right selectors for your reports. Here some usual ones: //APEX 5: table.a-IRR-table tr, table.uReportStandard tr //APEX 4: table.apexir_WORKSHEET_DATA tr, table.uReportStandard tr</code></pre> I give you no guarantee that this will also work with your theme. Please ask your browsers developer tools and inspect your report layouts ;-) Create a dynamic action for a double click on a jQuery selector (see examples above) and set the event scope to dynamic. The dynamic action needs to execute JavaScript: $(this.triggeringElement).find('td:first>a')[0].click();</code></pre> Happy coding :-) Ottmar

A complete example</h2>
For those who want to see how everything can work together: I took the examples here in the article from my open source project Oracle Instrumentation Console</a>.</p>

Create fast insert scripts

New Version of Oracle Table API Generator Available

How to migrate Oracle Cloud wallet into existing auto_login_local wallet

How to use mkstore and orapki with Oracle Instant Client

Sample Data Sets for Oracle

Why I Like SQL*Plus for Deployment Scripts

UPDATE July 20, 2020</h2>
The bug still exists in SQLcl 20.2 and SQL Developer 20.2 under Windows - we have to wait a bit longer…</p>

Download BLOBs with SQL*Plus

Quick start - version control for existing Oracle projects

New Major Version of PL/SQL Export Utilities Available

New Version of D3 Force Network Chart Available

PLOC - Convert PL/SQL Code into Markdown Docs

Table API Generator v0.5.0 Available

New Major Version of D3 Force Network Chart Available

PLEX - PL/SQL Export Utilities

Thats It</h2>
Hopefully PLEX is usefull for someone else.</p>
Happy coding, apexing, version controlling :-)</p>
Ottmar</p>

APEX_EXPORT and Version Control

Show Today Button in APEX Datepicker

HydeBar

Form Behaviour - Bug or Feature?

Oracle DB 11gXE Install File Swap Check Disabler

jQuery Plugin svg2img

Pitfalls with Oracle 11g XE and Docker on Mac OS

APEX Plugin dhtmlxGantt

DokuWiki Plugin Reveal.js

Markdown Reporter

Table API Generator

Markdown for Oracle APEX

Why a subdomain for an APEX development system is a bad idea

D3.js Force-Directed Network Chart

Double click on APEX report row opens first link target

Ottmar’s Notes

Caddy Server - APEX behind a fully automated SSL proxy #JoelKallmanDay

Generate APEX plug-ins on the command line

Yet another Oracle DB logging tool: Console

Ottmar’s Notes

Caddy Server - APEX behind a fully automated SSL proxy #JoelKallmanDay

ORDS Configuration</h2> Here is my sample configuration for the ORDS (settings.xml):</p>

Start Caddy</h2> If you have everything together and your Caddyfile is in the same directory where you start Caddy, then the start command is particularly short:</p>

Generate APEX plug-ins on the command line

JavaScript helper functions for conversion</h2> Since I implemented the file watcher and other scripts with Node.js/npm, it was obvious to solve this in JavaScript as well. First the needed modules and a md5 hash function:</p>

Yet another Oracle DB logging tool: Console

Create fast insert scripts

New Version of Oracle Table API Generator Available

Conclusion</h2> A big THANK YOU to all contributors - the current version of the generator would not be possible without your feedback and help.</p>

How to migrate Oracle Cloud wallet into existing auto_login_local wallet

How to use mkstore and orapki with Oracle Instant Client

Sample Data Sets for Oracle

List of Global Helper Scripts</h2> Disable/enable all foreign key constraints</li> Create missing foreign key indexes</li> Unify constraint names</li> Unify index names</li>

Why I Like SQL*Plus for Deployment Scripts

UPDATE July 20, 2020</h2> The bug still exists in SQLcl 20.2 and SQL Developer 20.2 under Windows - we have to wait a bit longer…</p>

Download BLOBs with SQL*Plus

Quick start - version control for existing Oracle projects

New Major Version of PL/SQL Export Utilities Available

New Version of D3 Force Network Chart Available

PLOC - Convert PL/SQL Code into Markdown Docs

Table API Generator v0.5.0 Available

New Major Version of D3 Force Network Chart Available

PLEX - PL/SQL Export Utilities

Inspirations / Further Reading</h2> Thanks are going to:</p> André Borngräber for his ability to think and discuss database topics in deep details</li>

Thats It</h2> Hopefully PLEX is usefull for someone else.</p> Happy coding, apexing, version controlling :-)</p> Ottmar</p>

APEX_EXPORT and Version Control

Show Today Button in APEX Datepicker

HydeBar

Form Behaviour - Bug or Feature?

Oracle DB 11gXE Install File Swap Check Disabler

jQuery Plugin svg2img

Pitfalls with Oracle 11g XE and Docker on Mac OS

APEX Plugin dhtmlxGantt

DokuWiki Plugin Reveal.js

Markdown Reporter

Table API Generator

Markdown for Oracle APEX

Why a subdomain for an APEX development system is a bad idea

D3.js Force-Directed Network Chart

Double click on APEX report row opens first link target

UPDATE July 20, 2020</h2>
The bug still exists in SQLcl 20.2 and SQL Developer 20.2 under Windows - we have to wait a bit longer…</p>

Thats It</h2>
Hopefully PLEX is usefull for someone else.</p>
Happy coding, apexing, version controlling :-)</p>
Ottmar</p>