NONMEM Users Network Archive

Hosted by Cognigen

Re: [NMusers] [Suggest] NONMEM adds UTF-8 support for csv files

From: Ruben Faelens <ruben.faelens_at_gmail.com>
Date: Wed, 6 Jun 2018 10:48:34 +0200

Dear Mark,

It is definitely possible to add functionality in PsN to remove the BoM of
a CSV file automatically.
I suggest you add it in lib/tool/modelfit.pm on line #2934 (sub
copy_model_and_input, search for invokations of the cp() function).
Either you program something manually to detect a BoM, or you use File::BOM
http://search.cpan.org/~mattlaw/File-BOM-0.15/lib/File/BOM.pm

Of course, the proper way would be to add a switch -remove_bom, but we can
let the true PsN wizards handle that.

Good luck! Please share the code with us, or (better!) perform a pull
request on github.

Kind regards,
Ruben Faelens

On Wed, Jun 6, 2018 at 12:04 AM Mark Tepeck <mark.tepeck_at_gmail.com> wrote:

> Hi Joe and Sebastien,
>
>
> Thank you for your feedback. Microsoft software is surely not my favorite
> platform; However, I have to stick with MS due to the policy for the
> organization computer I use.
>
>
> I agree with Sebastien that there are some easy workarounds. It is indeed
> easy on an individual basis, but it can be substantial efforts if
> accumulated for all NONMEM users. NONMEM has more than a decade history;
> why couldn’t it be made more robust, versatile and user-friendly?
>
>
> Even if the new feature requested doesn’t benefit every end-user
> immediately, it might be useful one day. UFT-8 BOM csv files are widely
> supported by csv viewers, which speaks for their value in some scenarios.
> Why couldn’t it be supported by NONMEM as well? The most successf=
ul
> software are typically those constantly evolving and adapting to meet use=
rs
> rather than those ignoring users feedback and calling them to make an
> adaption.
>
>
> Best,
>
>
> Mark
>
> On Tue, Jun 5, 2018 at 9:44 AM, STANDING, Joseph (GREAT ORMOND STREET
> HOSPITAL FOR CHILDREN NHS FOUNDATION TRUST) <joseph.standing_at_nhs.net>
> wrote:
>
>> Dear Mark,
>>
>> Here is another workaround: abandon the use of Microsoft Excel (and all
>> possible Microsoft products) since this behaviour - new "improved" versi=
on
>> that you paid for - actually makes your life harder. I have also
>> encountered problems with Excel helpfully changing date format or time
>> zones without being asked - great of you are collating data recorded in
>> different time zones. If you must visualise your data in a spreadsheet
>> just use libre office, or some other open source solution, but in genera=
l
>> all data manipulations should be done in R not in a spreadsheet, so any
>> changes to the data are traceable.
>>
>> About 5 years ago I wiped Windows, installed Linux, and have never looke=
d
>> back. RStudio, NONMEM, LaTex and all sensible software work just the sa=
me,
>> yet your computer now lasts much longer and runs faster because it is no=
t
>> clogged up with inefficient operating system software. If you are
>> submitting to one of those backward journals that does not provide a LaT=
ex
>> template, simply write your paper in Rmarkdown and output as a Word doc
>> (equations even come out OK these days).
>>
>> Much of the above goes for Mac operating system too unfortunately.
>> Upgrade the operating system and NONMEM stops working, students have had
>> terrible problems. If you must have a Mac because you like paying twice
>> the price for a metal case, wipe the operating system and put linux on. =
 I
>> can highly recommend Lubuntu.
>>
>> BW,
>>
>> Joe
>>
>>
>>
>>
>>
>> ________________________________________
>> From: owner-nmusers_at_globomaxnm.com [owner-nmusers_at_globomaxnm.com] on
>> behalf of Sebastien Bihorel [sebastien.bihorel_at_cognigencorp.com]
>> Sent: 04 June 2018 14:04
>> To: Mark Tepeck
>> Cc: nmusers_at_globomaxnm.com
>> Subject: Re: [NMusers] [Suggest] NONMEM adds UTF-8 support for csv files
>>
>> Hi,
>>
>> Two things:
>> - NONMEM does not require ID to be the first variable in $INPUT
>> - the headers of your csv file do not have to match the variable names
>> used in $INPUT
>>
>> So you have 2 easy workarounds for your issue:
>> - as previously pointed out, you can use another first variable... a
>> deletion flag variable or a study variable are as good candidates as ID =
for
>> the 1st column of your data.
>> - rename your ID variable as NMID in your csv file (or any alphanumeric
>> combo that do not start with ID for that matter)
>>
>> Sebastien
>>
>> ________________________________
>> From: "Mark Tepeck" <mark.tepeck_at_gmail.com>
>> To: "Nick Holford" <n.holford_at_auckland.ac.nz>, nmusers_at_globomaxnm.com
>> Sent: Monday, June 4, 2018 7:52:28 AM
>> Subject: Re: [NMusers] [Suggest] NONMEM adds UTF-8 support for csv files
>>
>> Hi All,
>>
>> Unfortunately, the latest release of MS Excel 2017 (build 9330.2087 for
>> office 365) becomes more restrictive on SYLK files, which is apparently =
a
>> step-back update, but really a typical Microsoft behavior ☹. Eve=
n you agree
>> to continue with the warning messages, the latest Excel will end up with=
 a
>> blank sheet. Using #ID is a practical workaround for this situation but
>> definitely not the most intuitive one. On the other hand, even most NONM=
EM
>> users are already used to some ‘tricks’, this issue stil=
l costs unnecessary
>> efforts and deepens the learning curve for NONMEM newbies. Personally, i=
t
>> is surprising to see a well-maintained software lack support for UTF-8 c=
sv
>> files, which are now well supported by all major csv viewers, e.g.
>> Editpad, Notepad++, open office, MS office. If this is a small fix for
>> NONMEM, I sincerely hope that it could become true. This feature might
>> potentially streamline NONMEM runs without adding # , as well as make
>> NONMEM learning more fun.
>>
>> Thank you,
>>
>> Mark
>>
>> On Mon, Jun 4, 2018 at 1:15 AM, Nick Holford <n.holford_at_auckland.ac.nz
>> <mailto:n.holford_at_auckland.ac.nz>> wrote:
>> Hi,
>> Excel can open files with ID as the first column header. You just have t=
o
>> agree to continue when Excel shows its warning messages.
>> Personally I use #ID as the first column header which avoids the Excel
>> warnings and also means I don’t need to specify an IGNORE charac=
ter in
>> NM-TRAN because # is the default.
>> Best wishes,
>> Nick
>>
>> --
>> Nick Holford, Professor Clinical Pharmacology
>> Dept Pharmacology & Clinical Pharmacology, Bldg 503 Room 302A
>> University of Auckland,85 Park Rd
>> <https://maps.google.com/?q=Auckland,85+Park+Rd&entry=gmail&source=
=g><
>> https://maps.google.com/?q=85+Park+Rd&entry=gmail&source=g>,Privat=
e Bag
>> 92019,Auckland,New Zealand
>> office:+64(9)923-6730 <+64%209-923%206730> mobile:NZ+64(21)46 23 53 FR+3=
3(6)62
>> 32 46 72 <+33%206%2062%2032%2046%2072>
>> email: n.holford_at_auckland.ac.nz<mailto:n.holford_at_auckland.ac.nz>
>> http://holford.fmhs.auckland.ac.nz/
>> http://orcid.org/0000-0002-4031-2514
>> Read the question, answer the question, attempt all questions
>>
>> From: owner-nmusers_at_globomaxnm.com<mailto:owner-nmusers_at_globomaxnm.com> =
<
>> owner-nmusers_at_globomaxnm.com<mailto:owner-nmusers_at_globomaxnm.com>> On
>> Behalf Of HUI, Ka Ho
>> Sent: Monday, June 4, 2018 4:48 AM
>> To: Mark Tepeck <mark.tepeck_at_gmail.com<mailto:mark.tepeck_at_gmail.com>>;
>> nmusers_at_globomaxnm.com<mailto:nmusers_at_globomaxnm.com>
>> Subject: [FORGED] [NMusers] 回覆: [NMusers] [Suggest] NONM=
EM adds UTF-8
>> support for csv files
>>
>> Dear Mark
>>
>> Does it work if you don't put "ID" as the first column? Back to the days
>> when I frequently used Excel, this had been the solution for me.
>>
>> Matthew
>>
>> ------ 原有訊息------
>> 寄件者: Mark Tepeck
>> 日期: 2018年6月4日週一 =
10:41
>> 收件者: nmusers_at_globomaxnm.com<mailto:nmuse=
rs_at_globomaxnm.com>;
>> 副本:
>> 主旨:[NMusers] [Suggest] NONMEM adds UTF-8 support=
 for csv files
>>
>> Hi NONMEM Users,
>>
>> As a PK scientist, I often manipulate csv files with Microsoft Excel, R,
>> and NONMEM. As some of you may have seen, if the first record in a csv f=
ile
>> is a string “ID”, Microsoft Excel would mistake it as an=
 SYLK file which
>> cannot be opened normally by Excel. One workaround is to use any strings
>> other than ID, e.g. id, '‘ID’. However, this may cause n=
ew problems in some
>> scenarios. Another potential solution is to use the R function
>> readr::write_excel_csv ('dataframe.csv') to output a csv file with UTF-8
>> Byte order mark. Unfortunately, this csv file turned out to become
>> unreadable for NONMEM although Excel likes it. Since waiting for bug fix=
ing
>> of Excel by Microsoft is not optimistic, I would truly appreciate it if
>> NONMEM can add UTF-8 support.
>>
>> Thank you,
>>
>> Mark Tepeck
>>
>>
>>
>>
>> ************************************************************************=
********************************************
>>
>> This message may contain confidential information. If you are not the
>> intended recipient please inform the
>> sender that you have received the message in error before deleting it.
>> Please do not disclose, copy or distribute information in this e-mail or
>> take any action in relation to its contents. To do so is strictly
>> prohibited and may be unlawful. Thank you for your co-operation.
>>
>> NHSmail is the secure email and directory service available for all NHS
>> staff in England and Scotland. NHSmail is approved for exchanging patien=
t
>> data and other sensitive information with NHSmail and other accredited
>> email services.
>>
>> For more information and to find out how you can switch,
>> https://portal.nhs.net/help/joiningnhsmail
>>
>>
>


Received on Wed Jun 06 2018 - 04:48:34 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request_at_iconplc.com. Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.